Skip to content

Azure mysql postgres auto discovery configuration#15629

Merged
GavinFrazar merged 16 commits into
masterfrom
gavinfrazar/azure_mysql_postgres_auto_discovery_configuration
Aug 19, 2022
Merged

Azure mysql postgres auto discovery configuration#15629
GavinFrazar merged 16 commits into
masterfrom
gavinfrazar/azure_mysql_postgres_auto_discovery_configuration

Conversation

@GavinFrazar
Copy link
Copy Markdown
Contributor

@GavinFrazar GavinFrazar commented Aug 17, 2022

This PR adds configuration for Azure database auto-discovery.

This is the first of several PRs to implement #14688

Configuration allows a user to filter by subscription, resource group, region, tags. If subscription or resource group is omitted, then all subscriptions or all resource_groups are queried. Users can also use * to specify this behavior explicitly. Example configs:

db_service:
  enabled: "yes"
  resources:
    - labels:
        "*": "*"
  azure:
    # discover all postgres DBs in all subscriptions and resource groups, in the eastus region.
    - subscriptions: ["*"]
      resource_groups: ["*"]
      types: ["postgres"]
      regions: ["eastus"]
      tags:
        "*": "*"
    # discover all mysql DBs in all subscriptions and resource groups, in the eastus or centralus.
    - types: ["mysql"]
      regions: ["eastus", "centralus"]
      tags:
        "*": "*"
    # discover all mysql and postgres DBs a specific subscription and resource group, in the eastus region
    - subscriptions: ["060a97ea-3a57-4218-9be5-dba3f19ff2b5"]
      resource_groups: ["database-access"]
      types: ["mysql", "postgres"]
      regions: ["eastus"]
      tags:
        "*": "*"

@GavinFrazar GavinFrazar changed the base branch from gavinfrazar/azure_mysql_postgres_auto_discovery to master August 17, 2022 21:56
@GavinFrazar GavinFrazar force-pushed the gavinfrazar/azure_mysql_postgres_auto_discovery_configuration branch from 44b51c5 to 94a66e2 Compare August 17, 2022 21:59
@GavinFrazar GavinFrazar force-pushed the gavinfrazar/azure_mysql_postgres_auto_discovery_configuration branch from 94a66e2 to 38e1df7 Compare August 17, 2022 22:05
@GavinFrazar
Copy link
Copy Markdown
Contributor Author

Force pushed to split out some unneeded code and restructure my commits.

@GavinFrazar GavinFrazar added feature-request Used for new features in Teleport, improvements to current should be #enhancements database-access Database access related issues and PRs db/postgres PostgreSQL related database access issues db/mysql MySQL related database access issues azure labels Aug 17, 2022
@GavinFrazar GavinFrazar marked this pull request as ready for review August 17, 2022 22:13
@github-actions github-actions Bot requested review from Tener and r0mant August 17, 2022 22:13
@GavinFrazar GavinFrazar changed the title Gavinfrazar/azure mysql postgres auto discovery configuration Azure mysql postgres auto discovery configuration Aug 17, 2022
Copy link
Copy Markdown
Contributor

@Tener Tener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as the configuration goes, apart from the small comments, I think this looks good.

It would be good to see the larger context still, i.e., actual db watcher code that would consume this configuration and execute the matching logic itself.

Comment thread lib/services/matchers.go Outdated
Comment thread lib/config/configuration.go Outdated
@smallinsky smallinsky requested a review from greedy52 August 18, 2022 11:06
Comment thread lib/config/configuration_test.go Outdated
Comment thread lib/config/fileconf.go Outdated
Comment thread lib/config/fileconf.go Outdated
Comment thread lib/services/matchers.go Outdated
Comment thread lib/config/configuration.go Outdated
Comment thread lib/config/configuration_test.go
Comment thread lib/config/fileconf.go Outdated
Comment thread lib/config/testdata_test.go
@codingllama
Copy link
Copy Markdown
Contributor

Just dropping by to say that the PR sizes are excellent now, thanks for the refactors @GavinFrazar. 🤌

You've got a healthy number of active reviewers, so I'll leave it to them.

GavinFrazar and others added 4 commits August 18, 2022 09:45
Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
@GavinFrazar GavinFrazar enabled auto-merge (squash) August 18, 2022 18:12
@GavinFrazar
Copy link
Copy Markdown
Contributor Author

@smallinsky friendly ping for re-review, since I need approval from both reviewers list. I think I addressed all of your concerns.

@GavinFrazar GavinFrazar enabled auto-merge (squash) August 19, 2022 19:05
@GavinFrazar GavinFrazar merged commit 951ee34 into master Aug 19, 2022
@GavinFrazar GavinFrazar deleted the gavinfrazar/azure_mysql_postgres_auto_discovery_configuration branch August 19, 2022 19:48
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
Azure mysql postgres auto discovery configuration (#15629)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery config create (#15630)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Add config create flags for azure matchers

* Add config create tests for azure

* Move discovery flags for azure below aws

* Fixup merge

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery config create (#15630)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Add config create flags for azure matchers

* Add config create tests for azure

* Move discovery flags for azure below aws

* Fixup merge

* Add Azure resource ID to protos (#15673)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update protobuf and fix database serialization

* Update azure database spec/status

* Change proto to use resource id string

* Fix database serialization test

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Aug 31, 2022
* Azure mysql postgres auto discovery configuration (#15629)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery config create (#15630)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Add config create flags for azure matchers

* Add config create tests for azure

* Move discovery flags for azure below aws

* Fixup merge

* Azure API for DB discovery (#15674)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Go mod tidy to update dependencies

* Add azure response error conversion

* Check for azure access denied and give a helpful error message

* Add azure subscriptions api

* Add azure mysql/postgresql api and wrappers

* Test generic db server for azure

* Make server properties its own type

* Convert server types manually instead of via json

* Move server list method selection logic out of api client

* Update azure db server tests

* Fixup merge

* Update comments

* Update more comments and remove junk code

* Move all azure api into lib/cloud/azure

* Update state and version checks

* Add mutex to subscription client for caching, just in case

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/subscriptions_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Rename azure subscription client and remove sub ID caching

* Add reference links for azure db ports

* Move indirect dep into group

* Wrap all converted azure response errors

* Remove unreachable panic

* Godoc DBServer

* Remove maxPages arg to azure client funcs

* Gofmt

* Spacing between copyright and package

* import order

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Bump go.mod version to 1.18

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
GavinFrazar added a commit that referenced this pull request Sep 1, 2022
* Azure mysql postgres auto discovery configuration (#15629)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Update lib/services/matchers.go

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/config/fileconf.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/services/matchers.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Remove superfluous cmp option for diffing azure matcher

* Rename AzureMatchers Tags to ResourceTags

* Deduplicate subscription/resource groups and add tests

* Remove azure matcher config fixup

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery config create (#15630)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Add config create flags for azure matchers

* Add config create tests for azure

* Move discovery flags for azure below aws

* Fixup merge

* Azure API for DB discovery (#15674)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Go mod tidy to update dependencies

* Add azure response error conversion

* Check for azure access denied and give a helpful error message

* Add azure subscriptions api

* Add azure mysql/postgresql api and wrappers

* Test generic db server for azure

* Make server properties its own type

* Convert server types manually instead of via json

* Move server list method selection logic out of api client

* Update azure db server tests

* Fixup merge

* Update comments

* Update more comments and remove junk code

* Move all azure api into lib/cloud/azure

* Update state and version checks

* Add mutex to subscription client for caching, just in case

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/subscriptions_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Rename azure subscription client and remove sub ID caching

* Add reference links for azure db ports

* Move indirect dep into group

* Wrap all converted azure response errors

* Remove unreachable panic

* Godoc DBServer

* Remove maxPages arg to azure client funcs

* Gofmt

* Spacing between copyright and package

* import order

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Bump go.mod version to 1.18

* Azure MySQL and PostgreSQL DB Discovery (#15745)

* Add Azure auto-discovery configuration fields

* Init databases if azure matchers are in config

* Use AzureMatchers in db service

* Use all azure subscriptions/resource groups if omitted in matcher

* Add azure config tests

* Go mod tidy to update dependencies

* Add azure response error conversion

* Check for azure access denied and give a helpful error message

* Add azure subscriptions api

* Add azure mysql/postgresql api and wrappers

* Test generic db server for azure

* Make server properties its own type

* Convert server types manually instead of via json

* Move server list method selection logic out of api client

* Update azure db server tests

* Fixup merge

* Update comments

* Update more comments and remove junk code

* Move all azure api into lib/cloud/azure

* Update state and version checks

* Convert Azure DB Server into database

* Add mutex to subscription client for caching, just in case

* Test database conversion from azure db server

* Refactor common code

* Add azure matchers to watch config

* Remove unused imports

* Use common max pages in memorydb users

* Add azure clients to cloud clients

* Add azure fetchers to watcher

* Skip fetcher NotFound errors

* Add azure watcher tests

* Fix test comment

* Add wildcard region matching

* Remove redundant import

* Update lib/srv/db/cloud/watchers/azure.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Move db server listing logic into separate func

* Refactor some db conversion code

* Use local var to cache sub IDs

* Rename azure subscription client

* Rename azure subscription client and remove sub ID caching

* Move azure cloud clients into separate embedded interface

* Update watcher test for 'NotFound' handling

* Embed azureClients struct

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/subscriptions_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/cloud/azure/db_server_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Rename azure subscription client and remove sub ID caching

* Add reference links for azure db ports

* Move indirect dep into group

* Wrap all converted azure response errors

* Remove unreachable panic

* Godoc DBServer

* Remove maxPages arg to azure client funcs

* Gofmt

* Spacing between copyright and package

* import order

* Update lib/srv/db/cloud/watchers/azure.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Fix import order

* Refactor region matching

* Fix usage of azure clients to remove max pages arg

* Query azure subscriptions in fetcher.Get if matching wildcard subscription

* Fix azure matcher construction

* Test azure fetcher discovers new subscriptions

* Remove unused test helper func

* Set azure database resource id metadata

* Log errors when fetching from azure subscription fails

* Use more readable protocol in azure db description

* Get azure mysql engine version from labels

* Update lib/srv/db/cloud/watchers/watcher_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Update lib/srv/db/cloud/watchers/watcher_test.go

Co-authored-by: Marek Smoliński <marek@goteleport.com>

Co-authored-by: Marek Smoliński <marek@goteleport.com>

Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azure database-access Database access related issues and PRs db/mysql MySQL related database access issues db/postgres PostgreSQL related database access issues feature-request Used for new features in Teleport, improvements to current should be #enhancements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants