Azure MySQL and PostgreSQL DB Discovery by GavinFrazar · Pull Request #15745 · gravitational/teleport

GavinFrazar · 2022-08-23T01:36:39Z

This PR adds:

watchers/fetchers for Azure MySQL and PosgreSQL DB servers.
construction of types.Database from Azure MySQL/PostgreSQL DB servers.
skipping a fetcher inside Watcher.fetchAndSend() if the fetcher returns trace.NotFound.
- This way, resource groups which are not found do not prevent other fetchers from running.
Some refactoring to make MaxPages a common constant, which we were using in a few different places. I needed it for Azure as well.
wildcard * matching for regions, resource groups, subscriptions

This is the last (code) PR for the larger issue to implement #14688

It is branched off #15674 and targetting it as a base branch for merge, but github will retarget it to master automatically once the other branch merges.

PR updating documentation for our Azure MySQL/PostgreSQL guide will follow this PR.

…overy_api

…overy_watchers

GavinFrazar · 2022-08-29T05:33:02Z

@smallinsky @greedy52 I think this is ready for re-review

…overy_watchers

Co-authored-by: Marek Smoliński <marek@goteleport.com>

…overy_watchers

* Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Convert Azure DB Server into database * Add mutex to subscription client for caching, just in case * Test database conversion from azure db server * Refactor common code * Add azure matchers to watch config * Remove unused imports * Use common max pages in memorydb users * Add azure clients to cloud clients * Add azure fetchers to watcher * Skip fetcher NotFound errors * Add azure watcher tests * Fix test comment * Add wildcard region matching * Remove redundant import * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Move db server listing logic into separate func * Refactor some db conversion code * Use local var to cache sub IDs * Rename azure subscription client * Rename azure subscription client and remove sub ID caching * Move azure cloud clients into separate embedded interface * Update watcher test for 'NotFound' handling * Embed azureClients struct * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Fix import order * Refactor region matching * Fix usage of azure clients to remove max pages arg * Query azure subscriptions in fetcher.Get if matching wildcard subscription * Fix azure matcher construction * Test azure fetcher discovers new subscriptions * Remove unused test helper func * Set azure database resource id metadata * Log errors when fetching from azure subscription fails * Use more readable protocol in azure db description * Get azure mysql engine version from labels * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 * Azure MySQL and PostgreSQL DB Discovery (#15745) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Convert Azure DB Server into database * Add mutex to subscription client for caching, just in case * Test database conversion from azure db server * Refactor common code * Add azure matchers to watch config * Remove unused imports * Use common max pages in memorydb users * Add azure clients to cloud clients * Add azure fetchers to watcher * Skip fetcher NotFound errors * Add azure watcher tests * Fix test comment * Add wildcard region matching * Remove redundant import * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Move db server listing logic into separate func * Refactor some db conversion code * Use local var to cache sub IDs * Rename azure subscription client * Rename azure subscription client and remove sub ID caching * Move azure cloud clients into separate embedded interface * Update watcher test for 'NotFound' handling * Embed azureClients struct * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Fix import order * Refactor region matching * Fix usage of azure clients to remove max pages arg * Query azure subscriptions in fetcher.Get if matching wildcard subscription * Fix azure matcher construction * Test azure fetcher discovers new subscriptions * Remove unused test helper func * Set azure database resource id metadata * Log errors when fetching from azure subscription fails * Use more readable protocol in azure db description * Get azure mysql engine version from labels * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

GavinFrazar added 30 commits August 17, 2022 15:03

Add Azure auto-discovery configuration fields

f28ac40

Init databases if azure matchers are in config

ccf04e7

Use AzureMatchers in db service

eb631a8

Use all azure subscriptions/resource groups if omitted in matcher

053a2f9

Add azure config tests

38e1df7

Go mod tidy to update dependencies

fe933e1

Add azure response error conversion

f6fe1bc

Check for azure access denied and give a helpful error message

153e1be

Add azure subscriptions api

a27f74c

Add azure mysql/postgresql api and wrappers

abe6de5

Test generic db server for azure

38c399e

Make server properties its own type

a68a866

Convert server types manually instead of via json

8be2b95

Move server list method selection logic out of api client

3ada05f

Update azure db server tests

16d80ab

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

20623d8

…overy_api

Fixup merge

a15c78b

Update comments

ec0474b

Update more comments and remove junk code

6dc1fbd

Move all azure api into lib/cloud/azure

318841c

Update state and version checks

c0bfe73

Convert Azure DB Server into database

299029e

Add mutex to subscription client for caching, just in case

8a2ce5d

Test database conversion from azure db server

be5aa52

Refactor common code

07a2b64

Add azure matchers to watch config

6f4a87b

Remove unused imports

7ec17fe

Use common max pages in memorydb users

9f8438d

Add azure clients to cloud clients

8c68796

Add azure fetchers to watcher

7fae554

GavinFrazar and others added 5 commits August 26, 2022 16:32

Remove unused test helper func

f9d2eff

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

c4fadfd

…overy_watchers

Set azure database resource id metadata

edafd1c

Log errors when fetching from azure subscription fails

6942c08

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

6a5195a

…overy_watchers

greedy52 approved these changes Aug 29, 2022

View reviewed changes

Comment thread lib/services/database.go Outdated

Comment thread lib/services/database.go

GavinFrazar added 3 commits August 29, 2022 09:58

Use more readable protocol in azure db description

edb3ce5

Get azure mysql engine version from labels

b8167d3

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

3e325f4

…overy_watchers

smallinsky approved these changes Aug 30, 2022

View reviewed changes

Comment thread lib/srv/db/cloud/watchers/watcher.go Outdated

Comment thread lib/srv/db/cloud/watchers/watcher_test.go Outdated

Comment thread lib/srv/db/cloud/watchers/watcher_test.go Outdated

Comment thread lib/srv/db/cloud/watchers/watcher_test.go Outdated

github-actions Bot removed the request for review from codingllama August 30, 2022 11:07

GavinFrazar and others added 6 commits August 30, 2022 10:38

Update lib/srv/db/cloud/watchers/watcher_test.go

c410310

Co-authored-by: Marek Smoliński <marek@goteleport.com>

Update lib/srv/db/cloud/watchers/watcher_test.go

7e77a7a

Co-authored-by: Marek Smoliński <marek@goteleport.com>

Simplify matchers in watcher config check

9d1d541

Fix typo in test

0c017ee

Change test name

bb7b4ae

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

21fe4f3

…overy_watchers

GavinFrazar enabled auto-merge (squash) August 30, 2022 18:23

GavinFrazar added 2 commits August 30, 2022 11:23

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

bba95c1

…overy_watchers

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

457ba9b

…overy_watchers

GavinFrazar mentioned this pull request Aug 30, 2022

azure mysql postgres auto discovery docs #15986

Merged

GavinFrazar merged commit f02b30b into master Aug 30, 2022

GavinFrazar deleted the gavinfrazar/azure_mysql_postgres_auto_discovery_watchers branch August 30, 2022 23:42

GavinFrazar restored the gavinfrazar/azure_mysql_postgres_auto_discovery_watchers branch August 31, 2022 00:33

GavinFrazar mentioned this pull request Aug 31, 2022

[v10] azure mysql postgres auto discovery watchers #15992

Merged

GavinFrazar deleted the gavinfrazar/azure_mysql_postgres_auto_discovery_watchers branch August 31, 2022 02:34

GavinFrazar added the backport/branch/v10 label Sep 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure MySQL and PostgreSQL DB Discovery#15745

Azure MySQL and PostgreSQL DB Discovery#15745
GavinFrazar merged 86 commits into
masterfrom
gavinfrazar/azure_mysql_postgres_auto_discovery_watchers

GavinFrazar commented Aug 23, 2022 •

edited

Loading

Uh oh!

GavinFrazar commented Aug 29, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

GavinFrazar commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GavinFrazar commented Aug 29, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GavinFrazar commented Aug 23, 2022 •

edited

Loading