RFD - tsh upgrade#28337
Conversation
|
In the future please try to adhere to https://github.com/gravitational/teleport/blob/master/rfd/0000-rfds.md#process |
| One of our goals for Teleport Cloud is to increase the cadence at which clusters can be upgraded. | ||
| This requires that Teleport Agents and Teleport client tools be up to date with the latest compatible version of the cluster. |
There was a problem hiding this comment.
Is the automatic tsh upgrade process design for Cloud only or do we want to support self-hosted users as well?
There was a problem hiding this comment.
I think it should be supported for self-hosted as well. Doesn't feel like there would be much difference as the client will be checking the version against the proxy endpoint.
|
|
||
| Detecting the version of the Teleport cluster should be straight forward. | ||
| Teleport supports a ping RPC that returns a response containing information about the Teleport version of the cluster and the minimum client version required. | ||
| In the case that the cluster version of Teleport is newer than the tsh version, the user will be prompted to upgrade. |
There was a problem hiding this comment.
This implies there's no downgrade path? On one side, this mitigates downgrade attacks, on the other side, this means Cloud cannot perform an emergency rollback if things go sideways during a major update.
There was a problem hiding this comment.
Right, as mentioned above let's just have the client track the server version and upgrade/downgrade as needed.
| ### Identifying installation method | ||
|
|
||
| The Teleport client tools should be re-installed using the original installation method. | ||
| Teleport documents several different methods for installing the client tools. https://goteleport.com/docs/installation/ | ||
| - Teleport can be installed from source and built using go. This scenario can be handled the same as if Teleport was installed by downloading the pre-build binaries. | ||
| - Teleport pre-built binaries can be downloaded for the desired platform. | ||
| - Teleport can be installed through the system package managers. Teleport currently maintains DEB and RPM repositories. | ||
| - Teleport is available for installation using brew on macOS, but it is not officially supported by Teleport and not recommended. | ||
| - Teleport provides macOS installation packages. One package installs all Teleport binaries, while the other installs only the tsh client. | ||
| - Teleport provides a tsh only zip installation for windows. | ||
|
|
||
| Additionally, Teleport supports a couple different releases: Open Source, Enterprise, and a FIPS compliant release. | ||
| The upgrade process will need to detect which release was installed and make sure to re-install the same release. | ||
|
|
||
| In order to identify which installation method was used to install the Teleport client tools, various checks will be executed. | ||
| The default installation method will be downloading the pre-built binaries. |
There was a problem hiding this comment.
Having a different process per installation method sounds tedious. We'll have to maintain one installation/update method in tsh for each system/installation-method/package combination. Extensive testing will be hard and we'll have a lot of weird edge cases + we won't support custom package managers (hello brew/chocolatey).
Is there a way to unify the update process and make it distribution-agnostic? e.g. storing tsh binaries in the user's home and installing a launcher system-wide.
|
|
||
| ## What | ||
|
|
||
| A tsh subcommand that will prompt the user to upgrade if the version of tsh is older than the cluster version. |
There was a problem hiding this comment.
Should this RFD also cover tctl? From what I understand, Cloud users have to use a local tctl to perform administration operations.
If that's the case, will tctl upgrade work? Will tsh upgrade also update tctl?
There was a problem hiding this comment.
Yes, tctl should be in scope.
| $ sudo tsh upgrade | ||
| Teleport v12.0.0 | ||
| Proxy version v13.1.2 | ||
| Teleport client tools version is behind the Teleport cluster version, do you wish to upgrade? [y/N]: | ||
| no | ||
| Exiting without upgrading Teleport client tools. Teleport servers are compatible with clients that are on the same major version or one major version older. We recommend updating before Teleport client tools become incompatible. If you would like to upgrade manually, installation instructions can be found at https://goteleport.com/docs/installation/ |
There was a problem hiding this comment.
If I'm on a Debian server A, which is running teleport (an agent, or a full-featured Teleport cluster) and want to login to a Teleport instance running on a server B (let's say cloud). Cloud is running a different version than server A. What happens when I run sudo tsh update? Does it update the whole teleport package, including my teleport process running? Should it only update tsh? tsh and tctl?
| ## UX | ||
|
|
||
| The tsh upgrade command should simply upgrade the Teleport client tools while providing some basic observability about the currently installed version of the Teleport client tools, the Teleport cluster version, and what version the client tools will be upgraded to. | ||
| The upgrade command will require root privileges. |
There was a problem hiding this comment.
I suspect that many Cloud users might not have sudo/admin access on their workstations. Requiring sudo access might make us miss a lot of customers. Do we have a way to estimate what proportion of our users can run this command?
There was a problem hiding this comment.
If we just keep client cache somewhere we have access to, I don't think it'll need root?
| The tsh upgrade command should simply upgrade the Teleport client tools while providing some basic observability about the currently installed version of the Teleport client tools, the Teleport cluster version, and what version the client tools will be upgraded to. | ||
| The upgrade command will require root privileges. | ||
|
|
||
| If the Teleport client tools are up to date, simply output the client and server version and exit |
There was a problem hiding this comment.
What leads the user to run tsh upgrade? Users won't think to run tsh upgrade every week. Do we want to warn each time the versions are drifting too much? (what is the definition of "too much," lower than the min supported version, as soon as there's a drift?).
| ### Identifying cluster version | ||
|
|
||
| Detecting the version of the Teleport cluster should be straight forward. | ||
| Teleport supports a ping RPC that returns a response containing information about the Teleport version of the cluster and the minimum client version required. |
There was a problem hiding this comment.
Will pre-release teleport and/or tsh versions be handled differently?
| Teleport supports a ping RPC that returns a response containing information about the Teleport version of the cluster and the minimum client version required. | ||
| In the case that the cluster version of Teleport is newer than the tsh version, the user will be prompted to upgrade. | ||
|
|
||
| ### Identifying installation method |
There was a problem hiding this comment.
There's one more installation method for tsh – Teleport Connect comes bundled with it.
On Linux, the bundled version is automatically symlinked to /usr/local/bin/tsh if there's no existing symlink there. On Windows, the folder with the bundled tsh is automatically added to Path. On macOS you can select a menu option that adds tsh to path. See RFD 99 for more details.
If someone runs tsh upgrade using the binary bundled with Connect, I don't think we should upgrade the binary as it'll simply get overwritten on the next Connect update (which users have to do manually for now). But even more important, since Connect was first released we've always assumed that the version of Connect is bound to the version of tsh. As such, we didn't really put effort into making the gRPC service served by tsh daemon backwards compatible. So if someone updates the bundled tsh without updating Connect, they risk breaking the app.
I think for now when someone runs tsh upgrade we could detect if tsh is bundled with Connect. If so, instead of upgrading the binary we could show a message telling the user to download a new Connect version instead.
The bundled tsh lives in a specific folder in the installation folder of Connect, so for now we could see if simple pattern matching on the binary path would be sufficient.
|
|
||
| ## What | ||
|
|
||
| A tsh subcommand that will prompt the user to upgrade if the version of tsh is older than the cluster version. |
There was a problem hiding this comment.
Yes, tctl should be in scope.
| One of our goals for Teleport Cloud is to increase the cadence at which clusters can be upgraded. | ||
| This requires that Teleport Agents and Teleport client tools be up to date with the latest compatible version of the cluster. |
There was a problem hiding this comment.
I think it should be supported for self-hosted as well. Doesn't feel like there would be much difference as the client will be checking the version against the proxy endpoint.
| ## Details | ||
|
|
||
| The main idea here is to add an upgrade command to the Teleport client tools. This will detect the version of the Teleport cluster. | ||
| If the version of tsh is older than the version of the cluster, the user will be prompted to upgrade. |
There was a problem hiding this comment.
As we discussed in our sync, I think the criteria should be "if tsh/tctl version is different from the cluster it's talking to" so it would upgrade/downgrade itself as needed.
|
|
||
| Detecting the version of the Teleport cluster should be straight forward. | ||
| Teleport supports a ping RPC that returns a response containing information about the Teleport version of the cluster and the minimum client version required. | ||
| In the case that the cluster version of Teleport is newer than the tsh version, the user will be prompted to upgrade. |
There was a problem hiding this comment.
Right, as mentioned above let's just have the client track the server version and upgrade/downgrade as needed.
|
|
||
| ### Identifying installation method | ||
|
|
||
| The Teleport client tools should be re-installed using the original installation method. |
There was a problem hiding this comment.
We have discussed instead keeping a cache of tsh/tctl binaries of different versions somewhere on the client machine instead, right? Can you change this section to describe how that would work instead?
| #### Linux | ||
|
|
||
| Teleport currently maintains DEB and RPM repositores. | ||
| So before upgrading, the upgrade command needs to identify whether the system it's running on is a Debian or Red Hat distribution. | ||
| The upgrade command will look at the /etc/lsb-release or /etc/redhat-release files to identify the Linux distribution and to determine which package manager to use. | ||
|
|
||
| The installation method can be further narrowed down by using the package manager CLI tools dpkg for Debian, and rpm for Red Hat. | ||
| If Teleport was installed using the manager, dpkg or rpm can be used to identify which release was installed: teleport, teleport-ent, or teleport-ent-fips. | ||
| Once the release has been identified, the Teleport client tools can now be upgraded using the package manager. | ||
|
|
||
| If Teleport was not installed using the package manager, the Teleport client tools will be upgraded by downloading the pre-built binaries. | ||
|
|
||
| #### macOS | ||
|
|
||
| On macOS, the Teleport package is available for installation using the macOS package installer. | ||
| To detect whether or not Teleport was installed using the installer, the pkgutil command can be used to search for the Teleport package. | ||
| Only Teleport Open Source and Teleport Enterprise are supported on macOS. | ||
|
|
||
| A signed release of tsh is also supported on macOS and is required to use TouchID. | ||
| So tsh will need to be upgraded separately after the initial upgrade of the other Teleport client tools. | ||
|
|
||
| #### Windows | ||
|
|
||
| Only the tsh command is supported on Windows machines. | ||
| tsh only supports an Open Source release, so for windows machines, the tsh upgrade command simply needs to download the desired tsh binary and install it. |
There was a problem hiding this comment.
All of this (i.e. how Teleport was installed) will be irrelevant if we implement the client binary cache store we talked about. So tsh will basically become a thin shim that detects the cluster version and executes appropriate version tsh binary from the local cache (downloading it first if needed).
The question is how to upgrade the shim though.
| Only the tsh command is supported on Windows machines. | ||
| tsh only supports an Open Source release, so for windows machines, the tsh upgrade command simply needs to download the desired tsh binary and install it. | ||
|
|
||
| ## UX |
There was a problem hiding this comment.
Can you also describe the following things we discussed:
- Will the users be able to disable the auto-upgrade behavior? How? Describe tsh config changes.
- What will the default behavior be? We should default to auto-upgrade.
| ## UX | ||
|
|
||
| The tsh upgrade command should simply upgrade the Teleport client tools while providing some basic observability about the currently installed version of the Teleport client tools, the Teleport cluster version, and what version the client tools will be upgraded to. | ||
| The upgrade command will require root privileges. |
There was a problem hiding this comment.
If we just keep client cache somewhere we have access to, I don't think it'll need root?
|
|
||
| If the Teleport client tools are up to date, simply output the client and server version and exit | ||
| ```console | ||
| $ sudo tsh upgrade |
There was a problem hiding this comment.
Separate command is good, but it's "automatic upgrade" so tsh/tctl should also check the version on each command invocation and upgrade itself if needed "on demand". Can you cover this in the RFD as well and provide a UX example?
| * Engineering: ... | ||
| * Infra: ... | ||
| * Kube: ... |
There was a problem hiding this comment.
| * Engineering: ... | |
| * Infra: ... | |
| * Kube: ... | |
| * Engineering: @r0mant && @jimbishopp | |
| * Product: @klizhentas || @xinding33 | |
| * Security: @jentfoo |
|
Going to close this PR since I've let this PR go a little stale and I didn't follow the correct RFD process. I've created a new PR with an updated design at #30317. We've decided to limit the scope and only support Cloud users. This should simplify the design quite a bit. |
5 participants
Contributes to https://github.com/gravitational/cloud/issues/4880
tsh upgrade RFD