Add packaging utility for client tools auto updates by vapopov · Pull Request #47060 · gravitational/teleport

vapopov · 2024-10-01T19:34:02Z

In this PR implemented packaging utilities for teleport client tools binaries, implemented as part of the client tools auto updates #46587

sclevine

How and where is this packaging logic going to be used?

vapopov · 2024-10-01T21:21:52Z

How and where is this packaging logic going to be used?

@sclevine packaging only used for integration tests, also need to test unarchive logic in this PR

zmb3 · 2024-10-01T22:58:14Z

I think we already have some zip utilities in lib/utils somewhere.

It's fine if you want to create a new package but please consolidate old uses as well so we don't have multiple implementations of the same code.

zmb3 · 2024-10-01T23:00:02Z

+}
+
+// freeDiskWithReserve returns the available disk space.
+func freeDiskWithReserve(dir string) (uint64, error) {


I'm pretty sure we already have code that does this too (for reserving space for session recordings)

I couldn't find similar code to reuse for disk space check, we also need to support windows platform for this check.

I moved part of logic to utils directory. Regarding archive helpers in code what I found: we have in-memory helper and mostly direct usage of archive/tar, archive/zip, compress/gzip packages for compression or I'm missing something

For unarchive we have this one helper https://github.com/gravitational/teleport/blob/master/lib/utils/unpack.go#L39C1-L39C46 but seems like this one is unused code

We have a lib/utils/disk.go which contains a PercentUsed func. I would reuse or expand upon that (and at least put your disk space checking code in the same spot).

I actually moved this logic to disc.go and disc_windows.go

sclevine

@zmb3 curious if we have a better place for test helper packages? Or do we generally use utils for both test and implementation code?

zmb3 · 2024-10-03T21:22:38Z

@zmb3 curious if we have a better place for test helper packages? Or do we generally use utils for both test and implementation code?

If we have test-only code I'd prefer it to be in _test.go files so that it doesn't make its way into production binaries.

If that's not doable, everything in integration/helpers is test-only code.

CR changes

Replace creating directory with extract path as argument

rosstimothy · 2024-10-10T17:50:10Z

+	if stat.Bsize < 0 {
+		return 0, trace.Errorf("invalid size")
+	}


FYI @vapopov I'm seeing the following related to this change when running make lint-go locally:

lib/utils/disk.go:56:5: SA4003: no value of type uint32 is less than 0 (staticcheck) if stat.Bsize < 0 { ^

@rosstimothy thanks, I will change this one in next PR, previously it was unix package and replaced with syscall

ztypes_linux_arm.go:

type Statfs_t struct { Type int32 Bsize int32 ... }

actually syscall for linux also has similar types

type Statfs_t struct { Type int64 Bsize int64 }

* Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off (#49180) * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

* Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247)

* [v15] Client tools autoupdates (#48648) * Expose client tools auto update for find endpoint (#46785) * Expose client tools auto update for find endpoint * Group auto update settings in find response Log error instead returning error Add tests auto update settings in find endpoint Add check for not implemented error * Add more test cases * Client AutoUpdate proto structure changes (#47532) * Update client autoupdate proto structure * Replace with reserved * Fix unit tests * Add more info in proto * Rename proto to be aligned RFD namings * Replace enum type for ToolsMode to string * Add packaging utility for client tools auto updates (#47060) * Add packaging utility for client tools auto updates * Add error handling for close functions * Move archive to existing utils package * Move archive helpers to integration/helper CR changes * CR changes * CR changes * CR changes Replace creating directory with extract path as argument * CR changes * Validate full size before un-archive Extract files to extractDir with ignore dir structure * Change compressing with relative paths Add test for cleanup and fix skip logic * CR changes * CR changes * Fix linter * Client tools auto update (#47466) * Add client tools auto update * Replace fork for posix platform for re-exec Move integration tests to client tools specific dir Use context cancellation with SIGTERM, SIGINT Remove cancelable tee reader with context replacement Renaming * Fix syscall path execution Fix archive cleanup if hash is not valid Limit the archive write bytes * Cover the case with single package for darwin platform after v17 * Move updater logic to tools package * Move context out from the library Base URL renaming * Add more context in comments * Changes in find endpoint * Replace test http server with `httptest` Replace hash for bytes matching Proper temp file close for archive download * Add more context to comments * Move feature flag to main package to be reused * Constant rename * Replace build tag with lib/modules to identify enterprise build * Replace fips tag with modules flag * Client auto updates integration for {tctl,tsh} (#47815) * Client auto updates integration for tctl/tsh * Add version validation Fix recursive version check for darwin platform Fix cleanup for multi-package support * Fix identifying tools removal from home directory * Replace ToolsMode with ToolsAutoUpdate * Reuse insecure flag for tests * Fix CheckRemote with login * Fix windows administrative access requirement Update must be able to be canceled, re-execute with latest version or last updated Show progress bar before request is made * Fix update cancellation for login action Address review comments * Add signal handler with stack context cancellation * Use copy instead of hard link for windows Fix progress bar if we can't receive size of package * Replace with list in order to support manual cancel * Download archive package to temp directory * Decrease timeout for client tools proxy call * Add audit logs for auto update resources (#48218) * Connect: Make sure tsh auto-updates are turned off * Add dir for code shared between Node.js processes * Connect: Make sure tsh auto-updates are turned off * Pass TELEPORT_TOOLS_VERSION=off to tsh vnet-daemon * Disable client tools auto update disabled if there are no home dir (#49159) Move updater to general tools package * Move client auto update helper to lib package (#49247) * Restrict AutoUpdateVersion to be created/updated for cloud (#49008) (#50244) * Restrict AutoUpdateVersion to be created/updated for cloud * Check builtin Admin role and Cloud feature * More informative error message * Remove KindAutoUpdateAgentRollout from editor role preset * Add remove tctl command for AutoUpdateConfig and AutoUpdateVersion (#49532) (#49676) * Fix auto-update re-exec arguments modified by aliases (#50228) (#51183) * Fix auto-update re-exec arguments modified by aliases * Make arguments to be required to set * Restore progress bar show before request * Improve integration tests to execute with `tsh` and `tctl` Added a full-cycle integration test to verify client tools auto-updates within a test cluster by modifying AutoUpdateConfig and AutoUpdateVersion resources. The test executes the login command using alias configurations to ensure no recursive re-execution occurs. The updater binary used in integration tests has been replaced with the `Run` logic of tctl and tsh. * Set generated test password by env variable instead of constant value * Restore priority of env check over remote check In case of double re-execution case we should stop second one to prevent loop re-execution Drop localDir set during compilation * Add client tools auto update tctl commands (#47692) * Add client tools auto update tctl commands * Always print version for watch command Restrict update empty target version Rename command to upsert * Add alias on/off for tools mode Rename update command to configure * Add semantic version validation * Drop watch command for autoupdate * Replace Upsert with Update/Create Add format option for output json/yaml * Change update message * Use get/set naming for client-tools * Add mode to response * Change sub-command help messages Leave only aliases for enabled/disabled * Reorganize tctl commands to have commands not required auth client * Propagate insecure flag with global config to commands by context * Fix autoupdate command without auth client * Change commands to enable/disable/target * Add retry in case of the parallel request * Add more than one retry Code review changes * Add template for client tools auto-update download url (#51210) * Add templates for client tools auto-update download url * Change to base url setting by env MakeURL moved to common function to be general for both, agent and client tools * Add godoc for common function and constant for default package * Use flags and version arguments instead of revision * Move base url env to shared constant * Fix tests after backporting * Pass TELEPORT_TOOLS_VERSION=off when starting tshd * Prevent keystore cleanup to remove bin directory (#52331) * Don't show the progress bar until the request to the CDN is made * Fix Windows permission error with self remove (#52316) * Fix windows permission error with self replace * Aggregate errors * Update comments --------- Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>

Add packaging utility for client tools auto updates

e61c571

vapopov added backport/branch/v14 no-changelog Indicates that a PR does not require a changelog entry labels Oct 1, 2024

vapopov requested review from hugoShaka and sclevine October 1, 2024 19:34

github-actions Bot requested a review from camscale October 1, 2024 19:34

github-actions Bot added the size/md label Oct 1, 2024

sclevine reviewed Oct 1, 2024

View reviewed changes

Comment thread lib/utils/packaging/archive.go Outdated

Comment thread lib/utils/packaging/archive.go Outdated

Comment thread lib/utils/packaging/archive.go Outdated

Comment thread lib/utils/packaging/archive.go Outdated

zmb3 reviewed Oct 1, 2024

View reviewed changes

vapopov added 2 commits October 1, 2024 20:44

Add error handling for close functions

53c7b7c

Move archive to existing utils package

a3fc5e0

sclevine reviewed Oct 3, 2024

View reviewed changes

Comment thread lib/utils/packaging/unarchive.go Outdated

Comment thread lib/utils/packaging/unarchive.go Outdated

Comment thread lib/utils/archive.go Outdated

Comment thread lib/utils/packaging/unarchive_unix.go Outdated

Comment thread lib/utils/packaging/unarchive_unix.go Outdated

Move archive helpers to integration/helper

7892864

CR changes

sclevine reviewed Oct 3, 2024

View reviewed changes

Comment thread integration/helpers/archive.go

Comment thread lib/utils/packaging/unarchive.go Outdated

Comment thread lib/utils/packaging/unarchive.go

CR changes

b9f8db6

vapopov force-pushed the vapopov/updater-teleport-packaging branch from 5b5bca5 to b9f8db6 Compare October 3, 2024 23:47

sclevine reviewed Oct 4, 2024

View reviewed changes

CR changes

ea036ae

vapopov force-pushed the vapopov/updater-teleport-packaging branch from 817b230 to ea036ae Compare October 4, 2024 06:31

sclevine reviewed Oct 4, 2024

View reviewed changes

Comment thread integration/helpers/archive.go Outdated

Comment thread integration/helpers/archive.go Outdated

vapopov force-pushed the vapopov/updater-teleport-packaging branch from 396424e to 9273632 Compare October 4, 2024 20:06

camscale removed their request for review October 7, 2024 06:05

vapopov force-pushed the vapopov/updater-teleport-packaging branch from 9273632 to e81f6d0 Compare October 7, 2024 21:36

CR changes

0c76738

Replace creating directory with extract path as argument

vapopov force-pushed the vapopov/updater-teleport-packaging branch from e81f6d0 to 0c76738 Compare October 7, 2024 21:39

CR changes

f8bd0f9

sclevine approved these changes Oct 9, 2024

View reviewed changes

Comment thread lib/utils/packaging/unarchive_test.go Outdated

Comment thread lib/utils/packaging/unarchive_test.go Outdated

hugoShaka reviewed Oct 9, 2024

View reviewed changes

Comment thread integration/helpers/archive.go Outdated

CR changes

98b1f0a

hugoShaka approved these changes Oct 9, 2024

View reviewed changes

sclevine approved these changes Oct 9, 2024

View reviewed changes

vapopov removed backport/branch/v14 labels Oct 9, 2024

vapopov enabled auto-merge October 9, 2024 22:20

bernardjkim approved these changes Oct 9, 2024

View reviewed changes

vapopov disabled auto-merge October 9, 2024 22:47

vapopov added 2 commits October 9, 2024 18:49

Fix linter

eee173c

Merge branch 'master' into vapopov/updater-teleport-packaging

79d1a7d

vapopov enabled auto-merge October 9, 2024 23:09

vapopov added this pull request to the merge queue Oct 9, 2024

Merged via the queue into master with commit dfa9de1 Oct 9, 2024

vapopov deleted the vapopov/updater-teleport-packaging branch October 9, 2024 23:44

rosstimothy reviewed Oct 10, 2024

View reviewed changes

vapopov mentioned this pull request Oct 10, 2024

Client tools auto update #47466

Merged

vapopov mentioned this pull request Nov 8, 2024

[v16] Client tools autoupdates #48645

Merged

vapopov mentioned this pull request Nov 8, 2024

[v15] Client tools autoupdates #48648

Merged

Conversation

vapopov commented Oct 1, 2024

Uh oh!

sclevine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vapopov commented Oct 1, 2024

Uh oh!

zmb3 commented Oct 1, 2024

Uh oh!

zmb3 Oct 1, 2024

Choose a reason for hiding this comment

Uh oh!

vapopov Oct 3, 2024 • edited by zmb3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmb3 Oct 3, 2024

Choose a reason for hiding this comment

Uh oh!

vapopov Oct 3, 2024

Choose a reason for hiding this comment

Uh oh!

sclevine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zmb3 commented Oct 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rosstimothy Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vapopov Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vapopov Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vapopov Oct 3, 2024 •

edited by zmb3

Loading

rosstimothy Oct 10, 2024 •

edited

Loading

vapopov Oct 10, 2024 •

edited

Loading