Adding the Azure sync module functions along with new cloud client functionality by mvbrock · Pull Request #50366 · gravitational/teleport

mvbrock · 2024-12-17T22:52:23Z

This PR is a result of splitting #48843 into smaller PRs, and provides the Azure fetching functions and new cloud client functionality specifically for role assignments and definitions. This PR also incorporates the existing lib/msgraph client and removes the extraneous client code from the original PR.

github-actions · 2024-12-20T00:09:48Z

🤖 Vercel preview here: https://docs-hx7av502y-goteleport.vercel.app/docs

…rror group

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

…incipal

…nctionality (#50366) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Forgot to decouple role definitions fetching function from the fetcher * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Updating go.sum * Fixing rebase after protobuf gen * Nolinting until upstream PRs * Updating to use existing msgraph client * Adding protection around nil values * PR feedback * Updating principal fetching to incorporate metadata from principal subtypes * Updating opts to not leak URL parameters * Conformant package name * Using variadic options * PR feedback * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * Also returning expanded principals for improved readability * Removing ptrToList * PR feedback * Rebase go.sum stuff * Go mod tidy * Linting * Linting * Collecting errors from fetching memberships and using a WithContext error group * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * e ref update * Fixing method * Fetching group members from groups rather than memberships of each principal * Linting --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

* Adding Azure integration protobuf messages and gRPC methods (#48628) * Adding Azure integration gRPC messages and RPC methods * Make derive * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * More PR feedback and generating protobuf code * Make derive * Adding identities field to principals, condition to role assignments, and role name to role definitions * Rebase conflicts * Did not fully fetch from origin/master when rebasing * Removing azure config field and keeping poll_interval as-is * Correct from parent branch * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding doc comments to access graph proto * Adding object type to principals * Adding location to Azure virtual machines * Update proto/accessgraph/v1alpha/access_graph_service.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Moving Azure Discovery protobuf config to the Azure Discovery PR * Make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick grpc * Protobuf and configuration for Access Graph Azure Discovery (#50364) * Protobuf and configuration for Access Graph Azure Discovery * Adding godoc and removing Integration field from fileconf * Adding the Azure sync module functions along with new cloud client functionality (#50366) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Forgot to decouple role definitions fetching function from the fetcher * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Updating go.sum * Fixing rebase after protobuf gen * Nolinting until upstream PRs * Updating to use existing msgraph client * Adding protection around nil values * PR feedback * Updating principal fetching to incorporate metadata from principal subtypes * Updating opts to not leak URL parameters * Conformant package name * Using variadic options * PR feedback * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * Also returning expanded principals for improved readability * Removing ptrToList * PR feedback * Rebase go.sum stuff * Go mod tidy * Linting * Linting * Collecting errors from fetching memberships and using a WithContext error group * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * e ref update * Fixing method * Fetching group members from groups rather than memberships of each principal * Linting --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding Azure sync functionality which can be used by the Azure Fetcher (#50367) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Update sync process to use msgraph client * Conformant package name * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Update e ref * Linting * PR feedback * Adding test names to reconciliation tests * Adding channel buffer * Going back to just reading from channel * Linting * PR feedback * PR feedback * PR feedback * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Fixing flaky test * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Invoking the Azure fetcher in the Discovery service (#50369) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Linting * PR feedback * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Conformant package name * Removing msgraphclient (again?) * Rebase fixes * More cleanup * PR feedback --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding OIDC auth functionality to the Azure integration (#51219) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Protobuf gen fix * Rebase fixes * More cleanup * e ref update * Invoking token generation and returning the response * Quick test with a message to make sure RPC is invoked * Skeleton of new Azure OIDC RPC call * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Fixing mocks * Fix imports * Fix test * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * e ref * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Azure integration status reporting (#51391) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Rebase fixes * More cleanup * PR feedback * Invoking token generation and returning the response * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * Initial refactoring to include Azure status reporting * Converging status sync between AWS and Azure * Fixing test * Sending usage stats * Fix imports * Add godocs and correct a few comments * Removing the usage events for now --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick fixes * Azure integration command (#47541) * Initial command to create the managed identity and role * Adding permissions and applying command params * Adding graph permissions to the MSI * Updating parameters * Adding some details and cleaning up comments * Fixing go.sum * Linting * License * PR feedback * Decoupling sync config with an interface for testing * Tweaks to test mocking * PR feedback * Rebase adjustments * PR feedback * Switch to empty struct maps instead of bool maps for set representation * Godocs * Adding user agent to Azure SDK requests * Linting * Moving armcompute back to v3 * Post cherry-pick make grpc * Post rebase make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

…nctionality (gravitational#50366) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Forgot to decouple role definitions fetching function from the fetcher * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Updating go.sum * Fixing rebase after protobuf gen * Nolinting until upstream PRs * Updating to use existing msgraph client * Adding protection around nil values * PR feedback * Updating principal fetching to incorporate metadata from principal subtypes * Updating opts to not leak URL parameters * Conformant package name * Using variadic options * PR feedback * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * Also returning expanded principals for improved readability * Removing ptrToList * PR feedback * Rebase go.sum stuff * Go mod tidy * Linting * Linting * Collecting errors from fetching memberships and using a WithContext error group * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * e ref update * Fixing method * Fetching group members from groups rather than memberships of each principal * Linting --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

mvbrock changed the base branch from master to mvbrock/azure-integration-disco-proto December 17, 2024 23:42

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from a817501 to e88e148 Compare December 17, 2024 23:43

Base automatically changed from mvbrock/azure-integration-disco-proto to master December 18, 2024 20:24

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from e88e148 to 308bc2c Compare December 18, 2024 20:29

mvbrock mentioned this pull request Dec 18, 2024

Implement Azure resource fetching in the Discovery service #48843

Closed

mvbrock requested review from marcoandredinis and tigrato December 19, 2024 06:50

mvbrock marked this pull request as ready for review December 19, 2024 07:07

github-actions Bot requested a review from hugoShaka December 19, 2024 07:07

github-actions Bot added discovery size/lg labels Dec 19, 2024

mvbrock added the no-changelog Indicates that a PR does not require a changelog entry label Dec 19, 2024

tigrato reviewed Dec 19, 2024

View reviewed changes

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from 02bcb5e to 0a4c1f4 Compare December 19, 2024 23:59

mvbrock temporarily deployed to vercel December 19, 2024 23:59 — with GitHub Actions Inactive

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from 0a4c1f4 to 672ae8d Compare December 20, 2024 00:01

mvbrock requested a review from tigrato December 20, 2024 02:58

marcoandredinis reviewed Dec 20, 2024

View reviewed changes

Comment thread lib/msgraph/paginated.go Outdated

Comment thread lib/cloud/azure/roledefinitions.go

Comment thread lib/srv/discovery/fetchers/azure-sync/principals.go Outdated

mvbrock requested a review from marcoandredinis December 20, 2024 21:17

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from 2894408 to 0dc6820 Compare December 30, 2024 17:16

mvbrock commented Dec 30, 2024

View reviewed changes

Comment thread lib/msgraph/paginated.go Outdated

mvbrock requested a review from zmb3 December 30, 2024 17:51

zmb3 reviewed Dec 30, 2024

View reviewed changes

Comment thread lib/srv/discovery/fetchers/azure-sync/roledefinitions.go Outdated

Comment thread lib/srv/discovery/fetchers/azure-sync/roledefinitions.go Outdated

Comment thread lib/cloud/azure/roledefinitions.go

mvbrock mentioned this pull request Dec 31, 2024

Consolidating slice to pointer slice functions #50651

Merged

tigrato reviewed Jan 2, 2025

View reviewed changes

Comment thread lib/msgraph/paginated.go Outdated

tigrato reviewed Jan 2, 2025

View reviewed changes

Comment thread e

Comment thread lib/msgraph/paginated.go Outdated

Comment thread lib/msgraph/paginated.go Outdated

Comment thread lib/msgraph/paginated.go Outdated

mvbrock requested review from tigrato and zmb3 January 7, 2025 05:17

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from 932b2e8 to a20ac6f Compare January 7, 2025 20:19

mvbrock and others added 10 commits January 10, 2025 11:06

PR feedback

2b5b928

Rebase go.sum stuff

0572557

Go mod tidy

4959c11

Linting

8ac788c

Linting

194ae3e

Collecting errors from fetching memberships and using a WithContext e…

1eacf07

…rror group

Fixing go.mod

4cf25c4

Update lib/msgraph/paginated.go

ae6bba6

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

PR feedback

aafeb05

e ref update

8b3cd5d

mvbrock force-pushed the mvbrock/azure-integration-disco-azure branch from 8914c7d to 8b3cd5d Compare January 10, 2025 17:06

mvbrock added 2 commits January 10, 2025 11:08

e ref update

c2319c6

Fixing method

acf1415

mvbrock requested a review from tigrato January 10, 2025 17:28

mvbrock added 2 commits January 11, 2025 17:39

Fetching group members from groups rather than memberships of each pr…

3c90b4d

…incipal

Linting

c7cd683

tigrato approved these changes Jan 13, 2025

View reviewed changes

public-teleport-github-review-bot Bot removed the request for review from hugoShaka January 13, 2025 18:15

mvbrock added this pull request to the merge queue Jan 13, 2025

Merged via the queue into master with commit 47f4498 Jan 13, 2025

mvbrock deleted the mvbrock/azure-integration-disco-azure branch January 13, 2025 19:48

mvbrock mentioned this pull request Jan 13, 2025

Adding Azure sync functionality which can be used by the Azure Fetcher #50367

Merged

mvbrock mentioned this pull request Jan 31, 2025

[v17] Backport Azure integration in Discovery Service to v17 #51711

Closed

7 tasks

mvbrock mentioned this pull request Jan 31, 2025

[v17] Backport Azure integration in Discovery Service to v17 #51725

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the Azure sync module functions along with new cloud client functionality#50366

Adding the Azure sync module functions along with new cloud client functionality#50366
mvbrock merged 34 commits intomasterfrom
mvbrock/azure-integration-disco-azure

mvbrock commented Dec 17, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Dec 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mvbrock commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Dec 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mvbrock commented Dec 17, 2024 •

edited

Loading