Client tools auto update by vapopov · Pull Request #47466 · gravitational/teleport

vapopov · 2024-10-10T19:08:31Z

In this PR implemented logic for client tools auto update {tsh, tctl} described in this RFD #39805

Update logic downloads Teleport binaries from CDN, extracts data in tools sub-directory (going to be used $TELEPORT_HOME/bin/{UUID-update-pkg}/{tsh,tctl}) and creates symlinks for {tsh,tctl} to $TELEPORT_HOME/bin/{tsh,tctl}.
Added integration tests which compiles updater binary with update logic only to simulate command execution with requested version to downloads them from fake test CDN server.

vapopov · 2024-10-10T19:11:20Z

-		return 0, trace.Errorf("invalid size")
-	}
-	avail := stat.Bavail * uint64(stat.Bsize)
+	avail := uint64(stat.Bavail) * uint64(stat.Bsize)


This change related to this comment #47060 (comment)
also I'm locally checking linter for warnings related touint64(stat.Bavail)

Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming

Fix archive cleanup if hash is not valid Limit the archive write bytes

…-update

Base URL renaming

…-update

Replace hash for bytes matching Proper temp file close for archive download

hugoShaka

Looks good to me ~~but we might wanbt to check if there's a saner way to detect enterprise builds. I'll open a thread in #teleport-dev~~

Edit: just saw your comment

sclevine · 2024-10-18T18:54:17Z

@hugoShaka note lib/modules might not work for tbot: #47565 (comment)

vapopov · 2024-10-18T19:00:33Z

@@ -0,0 +1,25 @@
+//go:build fips


@zmb3 should I move this one to lib/modules to identify if the build supports the FIPS? Or there already a way to get this info from modules package

I think this is present as modules.GetModules().IsBoringCrypto() -- BoringCrypto is FIPS validated

hugoShaka · 2024-10-18T19:17:23Z

@hugoShaka note lib/modules might not work for tbot: #47565 (comment)

I think tbot should be considered an agent like teleport and be updated by the agent updater we're building (as opposed to tools self-update). According to the RFD we going to have a single agent updater for oss, ent, fips and it will figure out the right build from /find or /ping rather than any existing binary version or build flag. So I think everything will fit nicely?

sclevine · 2024-10-18T19:39:39Z

@hugoShaka tbot also has a oneshot mode for CI/CD use cases. The current (possibly only verbally discussed) plan is to allow tbot to be upgraded through either method. But this may not ship in the first release of client tools updates.

vapopov · 2024-10-18T19:41:03Z

@avatus @EdwardDowling would appreciate your review, need one more approval

* Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off (#49180) * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247)

* [v15] Client tools autoupdates (#48648) * Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) * Restrict AutoUpdateVersion to be created/updated for cloud (#49008) (#50244) * Restrict AutoUpdateVersion to be created/updated for cloud * Check builtin Admin role and Cloud feature * More informative error message * Remove KindAutoUpdateAgentRollout from editor role preset * Add remove tctl command for AutoUpdateConfig and AutoUpdateVersion (#49532) (#49676) * Fix auto-update re-exec arguments modified by aliases (#50228) (#51183) * Fix auto-update re-exec arguments modified by aliases * Make arguments to be required to set * Restore progress bar show before request * Improve integration tests to execute with `tsh` and `tctl` Added a full-cycle integration test to verify client tools auto-updates within a test cluster by modifying AutoUpdateConfig and AutoUpdateVersion resources. The test executes the login command using alias configurations to ensure no recursive re-execution occurs. The updater binary used in integration tests has been replaced with the `Run` logic of tctl and tsh. * Set generated test password by env variable instead of constant value * Restore priority of env check over remote check In case of double re-execution case we should stop second one to prevent loop re-execution Drop localDir set during compilation * Add client tools auto update tctl commands (#47692) * Add client tools auto update tctl commands * Always print version for watch command Restrict update empty target version Rename command to upsert * Add alias on/off for tools mode Rename update command to configure * Add semantic version validation * Drop watch command for autoupdate * Replace Upsert with Update/Create Add format option for output json/yaml * Change update message * Use get/set naming for client-tools * Add mode to response * Change sub-command help messages Leave only aliases for enabled/disabled * Reorganize tctl commands to have commands not required auth client * Propagate insecure flag with global config to commands by context * Fix autoupdate command without auth client * Change commands to enable/disable/target * Add retry in case of the parallel request * Add more than one retry Code review changes * Add template for client tools auto-update download url (#51210) * Add templates for client tools auto-update download url * Change to base url setting by env MakeURL moved to common function to be general for both, agent and client tools * Add godoc for common function and constant for default package * Use flags and version arguments instead of revision * Move base url env to shared constant * Fix tests after backporting * Pass TELEPORT_TOOLS_VERSION=off when starting tshd * Prevent keystore cleanup to remove bin directory (#52331) * Don't show the progress bar until the request to the CDN is made * Fix Windows permission error with self remove (#52316) * Fix windows permission error with self replace * Aggregate errors * Update comments --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

vapopov added the no-changelog Indicates that a PR does not require a changelog entry label Oct 10, 2024

github-actions Bot added the size/lg label Oct 10, 2024

github-actions Bot requested review from EdwardDowling and avatus October 10, 2024 19:09

vapopov commented Oct 10, 2024

View reviewed changes

vapopov requested review from hugoShaka and sclevine October 10, 2024 19:11

Add client tools auto update

ddcbaf9

vapopov force-pushed the vapopov/client-auto-update branch from 5bc4215 to ddcbaf9 Compare October 10, 2024 20:21

sclevine reviewed Oct 15, 2024

View reviewed changes

hugoShaka mentioned this pull request Oct 15, 2024

[teleport-update] Add enable command #47565

Merged

vapopov added 4 commits October 15, 2024 17:35

Replace fork for posix platform for re-exec

a3fa2c3

Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming

Fix syscall path execution

1eeb647

Fix archive cleanup if hash is not valid Limit the archive write bytes

Merge remote-tracking branch 'origin/master' into vapopov/client-auto…

ad30d61

…-update

Cover the case with single package for darwin platform after v17

d5dd770

vapopov force-pushed the vapopov/client-auto-update branch from 144d573 to d5dd770 Compare October 16, 2024 00:55

Move updater logic to tools package

5f5688e

vapopov force-pushed the vapopov/client-auto-update branch from 355c675 to 5f5688e Compare October 16, 2024 19:44

vapopov requested a review from sclevine October 16, 2024 19:44

sclevine reviewed Oct 16, 2024

View reviewed changes

vapopov added 2 commits October 16, 2024 17:24

Move context out from the library

892f6f7

Base URL renaming

Add more context in comments

bfb33f6

vapopov requested a review from sclevine October 16, 2024 22:27

vapopov added 2 commits October 17, 2024 11:32

Merge remote-tracking branch 'origin/master' into vapopov/client-auto…

d2b0d11

…-update

Changes in find endpoint

f74b472

sclevine reviewed Oct 17, 2024

View reviewed changes

Replace test http server with httptest

52e47c8

Replace hash for bytes matching Proper temp file close for archive download

vapopov requested a review from sclevine October 17, 2024 20:17

vapopov added 2 commits October 17, 2024 16:39

Add more context to comments

e635b85

Move feature flag to main package to be reused

b019206

vapopov force-pushed the vapopov/client-auto-update branch from c860415 to b019206 Compare October 17, 2024 20:55

sclevine approved these changes Oct 17, 2024

View reviewed changes

Comment thread lib/autoupdate/feature.go Outdated

Comment thread lib/autoupdate/tools/utils.go Outdated

Constant rename

0e36699

hugoShaka reviewed Oct 18, 2024

View reviewed changes

Comment thread lib/autoupdate/feature_ent.go Outdated

hugoShaka approved these changes Oct 18, 2024

View reviewed changes

Replace build tag with lib/modules to identify enterprise build

b06bf9c

vapopov commented Oct 18, 2024

View reviewed changes

Replace fips tag with modules flag

412124e

avatus approved these changes Oct 18, 2024

View reviewed changes

public-teleport-github-review-bot Bot removed the request for review from EdwardDowling October 18, 2024 19:48

vapopov added this pull request to the merge queue Oct 18, 2024

Merged via the queue into master with commit 84719b3 Oct 18, 2024

vapopov deleted the vapopov/client-auto-update branch October 18, 2024 20:53

vapopov mentioned this pull request Nov 8, 2024

[v16] Client tools autoupdates #48645

Merged

vapopov mentioned this pull request Nov 8, 2024

[v15] Client tools autoupdates #48648

Merged

Conversation

vapopov commented Oct 10, 2024

Uh oh!

vapopov Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hugoShaka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sclevine commented Oct 18, 2024

Uh oh!

vapopov Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

sclevine Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugoShaka commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sclevine commented Oct 18, 2024

Uh oh!

vapopov commented Oct 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hugoShaka left a comment •

edited

Loading

sclevine Oct 18, 2024 •

edited

Loading

hugoShaka commented Oct 18, 2024 •

edited

Loading