Azure mysql postgres auto discovery configuration#15629
Merged
GavinFrazar merged 16 commits intoAug 19, 2022
Merged
Conversation
44b51c5 to
94a66e2
Compare
94a66e2 to
38e1df7
Compare
Contributor
Author
|
Force pushed to split out some unneeded code and restructure my commits. |
Tener
approved these changes
Aug 18, 2022
smallinsky
reviewed
Aug 18, 2022
greedy52
reviewed
Aug 18, 2022
Contributor
|
Just dropping by to say that the PR sizes are excellent now, thanks for the refactors @GavinFrazar. 🤌 You've got a healthy number of active reviewers, so I'll leave it to them. |
Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
greedy52
approved these changes
Aug 18, 2022
Contributor
Author
|
@smallinsky friendly ping for re-review, since I need approval from both reviewers list. I think I addressed all of your concerns. |
smallinsky
approved these changes
Aug 19, 2022
…overy_configuration
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
* Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
* Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Add Azure resource ID to protos (#15673) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update protobuf and fix database serialization * Update azure database spec/status * Change proto to use resource id string * Fix database serialization test Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar
added a commit
that referenced
this pull request
Sep 1, 2022
* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 * Azure MySQL and PostgreSQL DB Discovery (#15745) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Convert Azure DB Server into database * Add mutex to subscription client for caching, just in case * Test database conversion from azure db server * Refactor common code * Add azure matchers to watch config * Remove unused imports * Use common max pages in memorydb users * Add azure clients to cloud clients * Add azure fetchers to watcher * Skip fetcher NotFound errors * Add azure watcher tests * Fix test comment * Add wildcard region matching * Remove redundant import * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Move db server listing logic into separate func * Refactor some db conversion code * Use local var to cache sub IDs * Rename azure subscription client * Rename azure subscription client and remove sub ID caching * Move azure cloud clients into separate embedded interface * Update watcher test for 'NotFound' handling * Embed azureClients struct * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Fix import order * Refactor region matching * Fix usage of azure clients to remove max pages arg * Query azure subscriptions in fetcher.Get if matching wildcard subscription * Fix azure matcher construction * Test azure fetcher discovers new subscriptions * Remove unused test helper func * Set azure database resource id metadata * Log errors when fetching from azure subscription fails * Use more readable protocol in azure db description * Get azure mysql engine version from labels * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds configuration for Azure database auto-discovery.
This is the first of several PRs to implement #14688
Configuration allows a user to filter by subscription, resource group, region, tags. If subscription or resource group is omitted, then all subscriptions or all resource_groups are queried. Users can also use
*to specify this behavior explicitly. Example configs: