[v10] Add AWS Roles to the buildbox pipeline#17295
Closed
wadells wants to merge 730 commits into
Closed
Conversation
Fix link in Authentication options docs Co-authored-by: Vitor Enes <vitor.duarte@goteleport.com>
Organize docs guide sections chronologically Backports #15357 * Organize docs guide sections chronologically This change aims to make docs navigation easier by organizing some docs sections according to the sequence of steps a user would take to set up Teleport. The current docs organization uses a variety of categories and schemes to organize the docs. For example, there is a "Home" section that includes the Changelog, Installation page, and Getting Started guides; a "Setup" section that includes references and admin guides; and edition-specific sections (Enterprise, Cloud). For a user who is setting up Teleport--or who has already done some setup work and wants more advanced instructions--it's difficult to know where in the docs to find the right information. This change organizes our how-to guides into the following categories that describe the process of setting up Teleport: - Try out Teleport - Deploy a Cluster (including choosing an edition) - Configure Access (including SSO, RBAC, and Access Requests) - Manage your Cluster (admin guides, operations, etc.) - Use Teleport (this section already exists) I moved the Reference section after this chronology, since users can access the reference guides anywhere in the setup process. As part of the change, I have also moved the content from the "Enterprise" and "Cloud" sections into "Deploy a Cluster", since this content has to do with how to deploy a specific edition of Teleport. Note that this change does _not_ attempt to reorganize our protocol-specific sections. While adding resources is part of the Teleport setup process, we have a lot of content in our protocol-specific sections, and moving it all into a single section related to adding resources to a cluster would (a) exceed the maximum depth for subsections in the nav bar and (b) cause more confusion than it alleviates. * Respond to PR feedback - Create a "Compliance Frameworks" section of "Configure Access" with the FedRAMP and SOC 2 guides - Rename "Use Teleport" to "Connect your Client" - Move the database GUI client guide into "Connect your Client" * Add redirects * Fix linter issues
Fix race in reversetunnel.remoteConn
Record when a session recording is accessed This adds a new SessionRecordingAccess event that is emitted every time a session's events are queried, or if a session's events are streamed. This event is emitted by both v1 and v2 of the API, so both the web UI and tsh play will result in an event in the audit log. Implements #13880. The frontend change for supporting this event type is here - gravitational/webapps#970.
Add default debug setting for install.sh AMI script.
Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
Apply linters to legacy protos (#15879) Applies linters to legacy protos and adds a few additional Makefile targets to make it easier to manage protos locally. Proto linters now run in CI. #15187 Backport #15879 to branch/v10 * Apply linters to legacy protos * Handle new folders in genproto.sh, reset gen/proto if exists * Lint and format lib/teleterm as part of protos/all
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Add Azure resource ID to protos (#15673) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update protobuf and fix database serialization * Update azure database spec/status * Change proto to use resource id string * Fix database serialization test Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
NodeJoin script: fix when no labels are provided
Recently we added a way to add labels on newly added nodes based on the
token.
Each token now has a list of SuggestedLabels, which are used to feed
that list.
However, if that list is empty, the generated script would trigger the
following error:
`teleport: error: unexpected`
This happens when running the `teleport node configure ...` command.
This happens because the command is generating an empty argument `""`
when running the `teleport node configure ...` command.
So it looks like this:
```bash
${TELEPORT_BINARY_DIR}/teleport node configure \
--token token \
joinmethod \
--ca-pin pin \
--auth-server host:port \
"" \
--output someport
```
That empty argument breaks things.
So, in order to fix it, we are going to change the default value when no
labels are provided.
Instead of an empty string, we'll use an empty array.
Demo (teleport node configure message removed for brev
No label
```bash
$ LABELS_FLAG=(); f=$(mktemp -d)/node.yaml; teleport node configure --auth-server w:1 "${LABELS_FLAG[@]}" --output $f && yq .s
sh_service.labels $f
enabled: "yes"
commands:
- name: hostname
command: [hostname]
period: 1m0s
```
Single label
```bash
$ LABELS_FLAG=(--labels x=y); f=$(mktemp -d)/node.yaml; teleport node configure --auth-server w:1 "${LABELS_FLAG[@]}" --output $f && yq .ssh_service $f
enabled: "yes"
labels:
x: "y"
commands:
- name: hostname
command: [hostname]
period: 1m0s
```
Multiple labels
```bash
$ LABELS_FLAG=(--labels x=y,dev=prod); f=$(mktemp -d)/node.yaml; teleport node configure --auth-server w:1 "${LABELS_FLAG[@]}" --output $f && yq .ssh_service $f
enabled: "yes"
labels:
dev: prod
x: "y"
commands:
- name: hostname
command: [hostname]
period: 1m0s
```
operator tests: fix flaky test
Update on-prem version to 10.1.9
* Add architecture guide for Machine ID * Adjust indentation per linter rule * Update docs/pages/machine-id/architecture.mdx Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Update docs/pages/machine-id/architecture.mdx Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Update docs/pages/machine-id/architecture.mdx Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Update docs/pages/machine-id/architecture.mdx Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Add section on joining and renewable certificates * spag * Add notes on bot creation, user and token. * Clarify bot destination configuration * Adjust indentation * Start fleshing out notes on file permissions * Finish filesystem permissions section * Update docs/pages/machine-id/architecture.mdx Co-authored-by: Tim Buckley <tim@goteleport.com> * Address Tim's PR comments * spelling correction * Apply suggestions from code review Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Address a few docs structure comments * Furth restructure architectural guide * spag * Avoid using the "static" terminology in reference to tokens. * appease linter no-heading-punctuation * Use paragraphs rather than ordered list for tbot actions * Add notes on "destinations" * Improve the way that the tbot section reads * Add notes on daemon vs oneshot. Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> Co-authored-by: Tim Buckley <tim@goteleport.com>
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 * Azure MySQL and PostgreSQL DB Discovery (#15745) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Convert Azure DB Server into database * Add mutex to subscription client for caching, just in case * Test database conversion from azure db server * Refactor common code * Add azure matchers to watch config * Remove unused imports * Use common max pages in memorydb users * Add azure clients to cloud clients * Add azure fetchers to watcher * Skip fetcher NotFound errors * Add azure watcher tests * Fix test comment * Add wildcard region matching * Remove redundant import * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Move db server listing logic into separate func * Refactor some db conversion code * Use local var to cache sub IDs * Rename azure subscription client * Rename azure subscription client and remove sub ID caching * Move azure cloud clients into separate embedded interface * Update watcher test for 'NotFound' handling * Embed azureClients struct * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Fix import order * Refactor region matching * Fix usage of azure clients to remove max pages arg * Query azure subscriptions in fetcher.Get if matching wildcard subscription * Fix azure matcher construction * Test azure fetcher discovers new subscriptions * Remove unused test helper func * Set azure database resource id metadata * Log errors when fetching from azure subscription fails * Use more readable protocol in azure db description * Get azure mysql engine version from labels * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
ListResources was adding the namespace to the backend key when retrieving WindowsDesktopServices, however UpsertWindowsDesktopService doesn't include the namespace in the prefix. This results in never getting any items from the backend. Removing the namespace in the prefix to match GetWindowsDesktopService revealed that we were also trying to unmarshal the resource as a types.Server. Switching the unmarshal function to use `services.UnmarshalWindowsDesktopService` as GetWindowsDesktopService allows ListResources to provide the correct resources. A new test case for WindowsDesktops was added to `local.TestListResources` to prevent any regressions.
The cacert flag was removed from the curl output during the tsh app login as most production Teleport clusters are likely to be using publicly trusted CAs, and therefore wouldn't need the flag. If the user specifies an insecure login, however, the cacert flag is included with the curl output as it used to be. Additionally, some tests have been added for the formatAppConfig function. It was discovered that the YAML output format was outputting two newlines, so a small modification was made to remove this. Addresses issue #7518.
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
) The periodic version metric calculation loaded all nodes, database servers and app servers into memory in order to tally the versions of each. For larger clusters unmarshalling all resources and loading them into memory is quite expensive. To prevent Teleport from potentiall being OOM killed we can use `ListNodes` and `ListResources` to limit the number of resources being loaded into memory.
This fixes a goroutine leak caused by the various connect methods getting stuck waiting to push their error in `errChan` after the function exited with a success.
Backport #17099 to branch/v10
A misbehaving server could send a single byte to the clipboard using format CF_UNICODETEXT, which would cause an underflow. Fixes gravitational/teleport-private#177
Co-authored-by: Hugo Shaka <hugo.hervieux@goteleport.com>
Without this any tag that isn't part of the history on master will fail to successfully promote. This breaks most dev builds, which don't end up as part of master or a release branch. (cherry picked from commit 531bc51)
…17167) * Dont try to print scripts in a table * Use Fprintf * Handle fprintf errors
…17226) Remote proxies do not require `types.KindInstaller`, `types.KindDatabase`, and `types.KindApp` watchers since they do not affect remote reverse tunnels tracked from leaf clusters. Fix incompatibility with pre/post `v10.2.1` versions as the `types.KindInstaller` feature did not exist in older versions. Fixes #17219
The blurb under joining nodes seemed awkwardly worded to me, so I've changed it.
* Flip the order of the quay and ecr pipelines These are reversed in master/v11 (ecr first, and then quay) and having the order consistent across branches will make future ports easier. * Add AWS roles to Drone pipelines Backports #17201 Contributes to gravitational/SecOps#213
These were broken, as I didn't realize that the build step needed AWS access to fetch the buildbox.
wadells
requested review from
klizhentas,
r0mant,
russjones and
zmb3
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports #17274