Adding Azure integration protobuf messages and gRPC methods by mvbrock · Pull Request #48628 · gravitational/teleport

mvbrock · 2024-11-07T20:22:50Z

Part of of https://github.com/gravitational/access-graph/issues/640 to provide visibility of Azure resource in the Access Graph, and originating from the Azure integration POC branch master...mvbrock/azure-integration-poc. This PR creates the protobuf messages and gRPC methods to transmit Azure resources from the Discovery service to the Access Graph, per https://github.com/gravitational/access-graph/issues/1312.

mvbrock · 2024-11-13T20:53:22Z

@tigrato I added a few more items to the proto spec:

identities to AzurePrincipals`, which contains sign-in identities that can be used to sign into the account (reference)
condition to AzureRoleAssignment, which specifies resource conditions on the role assignment (reference)
role_name to AzureRoleDefinition, which contains the role name as separate from the role definition name (reference)
data_actions and not_data_actions to AzureRBACPermission, to include permissions on data within a specific resources (reference)

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

… and role name to role definitions

tigrato

Looks good to me once the comments are addressed

tigrato · 2024-11-27T11:14:49Z

+  string subscription_id = 2;
+  google.protobuf.Timestamp last_sync_time = 3;
+  string display_name = 4;
+  repeated string member_of = 5;


what's this member of? are they groups, policies?

Groups and directories, per https://learn.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0#relationships

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

* Adding Azure integration gRPC messages and RPC methods * Make derive * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * More PR feedback and generating protobuf code * Make derive * Adding identities field to principals, condition to role assignments, and role name to role definitions * Rebase conflicts * Did not fully fetch from origin/master when rebasing * Removing azure config field and keeping poll_interval as-is * Correct from parent branch * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding doc comments to access graph proto * Adding object type to principals * Adding location to Azure virtual machines * Update proto/accessgraph/v1alpha/access_graph_service.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Moving Azure Discovery protobuf config to the Azure Discovery PR * Make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

* Adding Azure integration protobuf messages and gRPC methods (#48628) * Adding Azure integration gRPC messages and RPC methods * Make derive * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * More PR feedback and generating protobuf code * Make derive * Adding identities field to principals, condition to role assignments, and role name to role definitions * Rebase conflicts * Did not fully fetch from origin/master when rebasing * Removing azure config field and keeping poll_interval as-is * Correct from parent branch * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding doc comments to access graph proto * Adding object type to principals * Adding location to Azure virtual machines * Update proto/accessgraph/v1alpha/access_graph_service.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Moving Azure Discovery protobuf config to the Azure Discovery PR * Make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick grpc * Protobuf and configuration for Access Graph Azure Discovery (#50364) * Protobuf and configuration for Access Graph Azure Discovery * Adding godoc and removing Integration field from fileconf * Adding the Azure sync module functions along with new cloud client functionality (#50366) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Forgot to decouple role definitions fetching function from the fetcher * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Updating go.sum * Fixing rebase after protobuf gen * Nolinting until upstream PRs * Updating to use existing msgraph client * Adding protection around nil values * PR feedback * Updating principal fetching to incorporate metadata from principal subtypes * Updating opts to not leak URL parameters * Conformant package name * Using variadic options * PR feedback * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * Also returning expanded principals for improved readability * Removing ptrToList * PR feedback * Rebase go.sum stuff * Go mod tidy * Linting * Linting * Collecting errors from fetching memberships and using a WithContext error group * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * e ref update * Fixing method * Fetching group members from groups rather than memberships of each principal * Linting --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding Azure sync functionality which can be used by the Azure Fetcher (#50367) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Update sync process to use msgraph client * Conformant package name * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Update e ref * Linting * PR feedback * Adding test names to reconciliation tests * Adding channel buffer * Going back to just reading from channel * Linting * PR feedback * PR feedback * PR feedback * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Fixing flaky test * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Invoking the Azure fetcher in the Discovery service (#50369) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Linting * PR feedback * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Conformant package name * Removing msgraphclient (again?) * Rebase fixes * More cleanup * PR feedback --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding OIDC auth functionality to the Azure integration (#51219) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Protobuf gen fix * Rebase fixes * More cleanup * e ref update * Invoking token generation and returning the response * Quick test with a message to make sure RPC is invoked * Skeleton of new Azure OIDC RPC call * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Fixing mocks * Fix imports * Fix test * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * e ref * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Azure integration status reporting (#51391) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Rebase fixes * More cleanup * PR feedback * Invoking token generation and returning the response * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * Initial refactoring to include Azure status reporting * Converging status sync between AWS and Azure * Fixing test * Sending usage stats * Fix imports * Add godocs and correct a few comments * Removing the usage events for now --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick fixes * Azure integration command (#47541) * Initial command to create the managed identity and role * Adding permissions and applying command params * Adding graph permissions to the MSI * Updating parameters * Adding some details and cleaning up comments * Fixing go.sum * Linting * License * PR feedback * Decoupling sync config with an interface for testing * Tweaks to test mocking * PR feedback * Rebase adjustments * PR feedback * Switch to empty struct maps instead of bool maps for set representation * Godocs * Adding user agent to Azure SDK requests * Linting * Moving armcompute back to v3 * Post cherry-pick make grpc * Post rebase make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

…ional#48628) * Adding Azure integration gRPC messages and RPC methods * Make derive * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * More PR feedback and generating protobuf code * Make derive * Adding identities field to principals, condition to role assignments, and role name to role definitions * Rebase conflicts * Did not fully fetch from origin/master when rebasing * Removing azure config field and keeping poll_interval as-is * Correct from parent branch * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding doc comments to access graph proto * Adding object type to principals * Adding location to Azure virtual machines * Update proto/accessgraph/v1alpha/access_graph_service.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Moving Azure Discovery protobuf config to the Azure Discovery PR * Make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

mvbrock requested review from r0mant, ryanclark and tigrato November 7, 2024 20:22

github-actions Bot added the size/md label Nov 7, 2024

github-actions Bot requested review from Tener and kimlisa November 7, 2024 20:23

mvbrock added the no-changelog Indicates that a PR does not require a changelog entry label Nov 7, 2024

Tener approved these changes Nov 8, 2024

View reviewed changes

ryanclark reviewed Nov 8, 2024

View reviewed changes

Comment thread proto/accessgraph/v1alpha/azure.proto Outdated

Comment thread proto/accessgraph/v1alpha/azure.proto

tigrato reviewed Nov 8, 2024

View reviewed changes

mvbrock requested review from ryanclark and tigrato November 8, 2024 16:54

mvbrock mentioned this pull request Nov 12, 2024

Implement Azure resource fetching in the Discovery service #48843

Closed

mvbrock marked this pull request as draft November 13, 2024 16:55

mvbrock marked this pull request as ready for review November 13, 2024 20:22

github-actions Bot requested review from atburke and rosstimothy November 13, 2024 20:22

mvbrock force-pushed the mvbrock/azure-integration-grpc branch from ba0b726 to fe10587 Compare November 13, 2024 20:30

mvbrock force-pushed the mvbrock/azure-integration-grpc branch 2 times, most recently from 6c2c286 to 7d10b7c Compare November 14, 2024 19:52

mvbrock and others added 9 commits November 18, 2024 14:38

Adding Azure integration gRPC messages and RPC methods

fe461a4

Make derive

c8dec2f

Update proto/accessgraph/v1alpha/azure.proto

24574f5

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

Update proto/accessgraph/v1alpha/azure.proto

8c06206

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

Update proto/accessgraph/v1alpha/azure.proto

67436f3

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

PR feedback

d1b54f8

More PR feedback and generating protobuf code

efc2705

Make derive

2ebad31

Adding identities field to principals, condition to role assignments,…

2603345

… and role name to role definitions

mvbrock added 3 commits November 18, 2024 14:38

Rebase conflicts

9e17af7

Did not fully fetch from origin/master when rebasing

37cecd7

Removing azure config field and keeping poll_interval as-is

61e626b

mvbrock force-pushed the mvbrock/azure-integration-grpc branch from 7d10b7c to 61e626b Compare November 18, 2024 20:38

Correct from parent branch

e100c19

tigrato reviewed Nov 27, 2024

View reviewed changes

mvbrock and others added 4 commits November 27, 2024 10:16

Apply suggestions from code review

9800be4

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

PR feedback

fe2cbc5

Adding doc comments to access graph proto

b5ca748

Adding object type to principals

2ff1a65

mvbrock requested a review from tigrato November 27, 2024 19:44

rosstimothy removed their request for review November 27, 2024 20:12

Adding location to Azure virtual machines

4b1dfb5

tigrato approved these changes Nov 28, 2024

View reviewed changes

Comment thread api/proto/teleport/legacy/types/types.proto Outdated

Comment thread proto/accessgraph/v1alpha/access_graph_service.proto Outdated

Update proto/accessgraph/v1alpha/access_graph_service.proto

5764fb1

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

public-teleport-github-review-bot Bot removed request for atburke, kimlisa and r0mant December 2, 2024 16:47

mvbrock and others added 3 commits December 2, 2024 10:58

Moving Azure Discovery protobuf config to the Azure Discovery PR

96e6802

Make grpc

9f37406

Merge branch 'master' into mvbrock/azure-integration-grpc

b695a16

mvbrock added this pull request to the merge queue Dec 2, 2024

Merged via the queue into master with commit 4b55d7b Dec 2, 2024

mvbrock deleted the mvbrock/azure-integration-grpc branch December 2, 2024 18:00

mvbrock mentioned this pull request Jan 31, 2025

[v17] Backport Azure integration in Discovery Service to v17 #51725

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Azure integration protobuf messages and gRPC methods#48628

Adding Azure integration protobuf messages and gRPC methods#48628
mvbrock merged 22 commits intomasterfrom
mvbrock/azure-integration-grpc

mvbrock commented Nov 7, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvbrock commented Nov 13, 2024

Uh oh!

tigrato left a comment

Uh oh!

Uh oh!

Uh oh!

tigrato Nov 27, 2024

Uh oh!

mvbrock Nov 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mvbrock commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvbrock commented Nov 13, 2024

Uh oh!

tigrato left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tigrato Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

mvbrock Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mvbrock commented Nov 7, 2024 •

edited

Loading