xds: generic xDS client transport channel and ads stream implementation by purnesh42H · Pull Request #8144 · grpc/grpc-go

purnesh42H · 2025-03-05T14:01:31Z

This is the xDS transport channel implementation change that setup an ADS (Aggregated Data Service) stream with xDS management server which can be used by generic xDS client to do resource watching. The ADS stream and xDS channel uses the types and interfaces that were created in #8042. The e2e tests are written using types and interfaces in #8042 and grpctransport created in #8066.

The PR copies the existing ADS stream and channel implementation from internal xdsclient code along with the helpers that are part of gRPC and then modify them to use the generic client types and interfaces. The PR contains 6 commits. Odd commits (1, 3, 5) are doing the exact copy and even commits (2, 4, 6) are modifying their predecessor commit. The copy index is as follows

Copied Implementation (modified to use generic client configs and interfaces)

From	To
`xds/internal/xdsclient/transport/ads`	`xds/internal/clients/xdsclient/internal/ads`
`xds/internal/xdsclient/channel.go`	`xds/internal/clients/xdsclient/xdsclient/channel.go`
`xds/internal/xdsclient/channel_test.go`	`xds/internal/clients/xdsclient/xdsclient/channel_test.go`

Copied helpers common (mainly for testing and logging. only what is needed)

From	To
`internal/backoff`	`xds/internal/clients/internal/backoff`
`internal/buffer`	`xds/internal/clients/internal/buffer`
`internal/pretty`	`xds/internal/clients/internal/pretty`
`internal/testutils/xds/e2e`	`xds/internal/clients/internal/testutils/e2e`
`internal/testutils/xds/fakeserver`	`xds/internal/clients/internal/testutils/fakeserver`
`internal/testutils/channel.go`	`xds/internal/clients/internal/testutils/channel.go`

Copied helpers xdsclient (mainly for testing and logging. only what is needed)

From	To
`xds/internal/xdsclient/xdsresource/errors.go`	`xds/internal/clients/xdsclient/internal/xdsresource/errors.go`
`xds/internal/xdsclient/xdsresource/name.go`	`xds/internal/clients/xdsclient/internal/xdsresource/name.go`
`xds/internal/xdsclient/xdsresource/type.go`	`xds/internal/clients/xdsclient/internal/xdsresource/type.go`
`xds/internal/xdsclient/xdsresource/version/version.go`	`xds/internal/clients/xdsclient/internal/xdsresource/version.go`

RELEASE NOTES: None

codecov · 2025-03-05T14:40:53Z

Codecov Report

Attention: Patch coverage is 69.38588% with 334 lines in your changes missing coverage. Please review.

Project coverage is 82.07%. Comparing base (af07815) to head (78fe33f).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
xds/internal/clients/xdsclient/ads_stream.go	74.23%	84 Missing and 26 partials ⚠️
...nal/clients/xdsclient/internal/xdsresource/name.go	15.51%	49 Missing ⚠️
...al/clients/internal/testutils/fakeserver/server.go	60.39%	37 Missing and 3 partials ⚠️
xds/internal/clients/xdsclient/channel.go	77.12%	26 Missing and 9 partials ⚠️
xds/internal/clients/internal/pretty/pretty.go	0.00%	29 Missing ⚠️
.../internal/clients/internal/testutils/e2e/server.go	77.77%	15 Missing and 5 partials ⚠️
xds/internal/clients/internal/backoff/backoff.go	47.22%	18 Missing and 1 partial ⚠️
...clients/internal/testutils/restartable_listener.go	64.10%	13 Missing and 1 partial ⚠️
...internal/clients/internal/testutils/e2e/logging.go	25.00%	6 Missing ⚠️
...l/clients/xdsclient/internal/xdsresource/errors.go	58.33%	4 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8144      +/-   ##
==========================================
- Coverage   82.30%   82.07%   -0.24%     
==========================================
  Files         392      410      +18     
  Lines       39132    40251    +1119     
==========================================
+ Hits        32207    33034     +827     
- Misses       5603     5855     +252     
- Partials     1322     1362      +40

Files with missing lines	Coverage Δ
xds/internal/clients/internal/buffer/unbounded.go	`100.00% <100.00%> (ø)`
...l/clients/internal/syncutil/callback_serializer.go	`100.00% <100.00%> (ø)`
xds/internal/clients/internal/syncutil/event.go	`100.00% <100.00%> (ø)`
xds/internal/clients/internal/testutils/channel.go	`100.00% <100.00%> (ø)`
.../clients/internal/testutils/e2e/clientresources.go	`93.93% <93.93%> (ø)`
...nal/clients/xdsclient/internal/xdsresource/type.go	`50.00% <50.00%> (ø)`
...internal/clients/internal/testutils/marshal_any.go	`62.50% <62.50%> (ø)`
...l/clients/xdsclient/internal/xdsresource/errors.go	`58.33% <58.33%> (ø)`
...internal/clients/internal/testutils/e2e/logging.go	`25.00% <25.00%> (ø)`
...clients/internal/testutils/restartable_listener.go	`64.10% <64.10%> (ø)`
... and 7 more

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

linux-foundation-easycla · 2025-03-06T04:13:50Z

The committers listed above are authorized under a signed CLA.

✅ login: purnesh42H / name: Purnesh Dixit (300c5a7, 7aec21c, 33a2edb, 03196ed, fba2d82, 400a165, 36ffd55, d78beb4, 8a0900c, f647a58, 42faca1, a0dadb3, ba64568, 2b6c05c, 2216da2, de4587f, 123a5a5, c44b143, 241f2e1, e339ef4, 1d347b6, 78fe33f, a1cf9cf)

dfawley · 2025-03-07T21:04:00Z

 */

-// Package grpclog defines logging for grpc.
+// Package clientslog defines logging for clients.


We should see if there's something better to do for logging. What about slog in the stdlib?

Our only real constraint is that whatever we do needs to work with the existing grpclog stuff when we use these from grpc.

@easwars

Is this a blocker for PR or it can be tried after e2e working generic xDS client?

Estimate work needed to get a standard library like slog working

dfawley · 2025-03-07T21:09:52Z

-// This is kept in internal until the gRPC project decides whether or not to
+// This is kept in internal until the clients project decides whether or not to
 // allow alternative backoff strategies.
 package backoff


Global concern: are we sure we're not pulling in dead code as part of this? AIUI the existing dead code checker only considers unexported functions/symbols.

Maybe we can scale some of these things back to be simpler & smaller given the more limited usages.

I have pushed 2 commits of removing unused code from common helpers and xdsclient helpers respectively. That has reduced code by around 3000 lines. I think the helpers can still be simplified but I think that will require some refactoring/rewriting. Can we do that as separate PR once we have a working client?

If DefaultExponential is the only thing that is being used by the code, that should be the only thing exported. Everything else can be unexported.

Right. Unexported other structs.

dfawley · 2025-03-07T21:10:55Z

 /*
 *
- * Copyright 2019 gRPC authors.
+ * Copyright 2025 gRPC authors.


I'm not sure it's appropriate to change the copyright date on all these files. Let's just leave them alone.

So, these are new files because they are copied to new folder. So, won't they need new date?

dfawley · 2025-03-07T21:12:24Z

We should see if we really need this and kill it otherwise.

Its mainly for debugging. If we remove this, we can print the response or resource directly using %v

dfawley · 2025-03-07T21:15:02Z


 import (
-	iresolver "google.golang.org/grpc/internal/resolver"
+	iresolver "google.golang.org/grpc/xds/internal/clients/xdsclient/internal/testutils/resolver"


Why do we need name resolver stuff in here?

Removed as part of unused code

dfawley · 2025-03-07T21:22:44Z

+	ResourceTypeMapForTesting = make(map[string]any)
+	ResourceTypeMapForTesting[xdsresource.V3ListenerURL] = listenerType


This can be simplified:

ResourceTypeMapForTesting = map[string]any{xdsresource.V3ListenerURL: listenerType}

dfawley · 2025-03-07T21:23:38Z

None of this makes sense in the context of an xdsclient, and it should be removed. This is all grpc-specific definitions.

Removed as part of unused code

dfawley · 2025-03-07T21:25:42Z

 */

-package grpcsync
+package clientssync


Nit: maybe syncutil?

dfawley · 2025-03-07T21:29:54Z

+	// Note that this differs from FailedPrecondition. It indicates arguments
+	// that are problematic regardless of the state of the system
+	// (e.g., a malformed file name).
+	invalidArgumentErrCode = 3


Can you leverage existing symbols instead?

https://pkg.go.dev/google.golang.org/genproto/googleapis/rpc/code

dfawley · 2025-03-07T21:32:03Z

+ *
 */

+// Package xdsclient contains implementation of the xDS client.


We have an xdsclient subdirectory inside xdsclient? And it's not internal?

A goal of this change is to add no new exported symbols at all, or to keep that set very small and well understood. That includes new packages not under internal.

yeah i had to move the implementation to a separate package from config structs to avoid cyclic dependency from internal packages which needs xdsclient config types. With this structure, to create a new client, user will have to import .....clients\xdsclient\xdsclient

danielzhaotongliu · 2025-03-07T23:37:21Z

 	infoW := io.Discard

-	logLevel := os.Getenv("GRPC_GO_LOG_SEVERITY_LEVEL")
+	logLevel := os.Getenv("CLIENTS_GO_LOG_SEVERITY_LEVEL")


In general, what would be the migration strategy here for customers/users internally and externally?

All users would have to update to this new environment variable in order to preserve the user-specified log levels. Would this be fine and/or feasible? Similarly, for all other environment variable changes.

Would there be an overall migration/import plan and strategy?

This is a completely new folder in the grpc-go. Eventually this entire code is planned to move to a new repo. There is no change being done to existing code of internal xds client implementation so unless the user decides to switch to this new implementation, they will not notice a change.

So both versions would co-exist for some time? If so, how long until we move the default to the new implementation?

Perhaps an overall migration and/or user adoption strategy should be documented in the design doc?

Actually, there are 2 things to note. If you are using internal xds client and don't have a requirement of not having grpc and proto dependencies, you can continue to use the internal xds client and you should not notice any behaviour change. However, if you want to configure your own transport and can't use proto based API(s), it would be good to switch to generic xDS client.

So both versions would co-exist for some time? If so, how long until we move the default to the new implementation?

it has to be fast follow up. Not more than 2-3 weeks of generic xDS client completion.

danielzhaotongliu · 2025-03-07T23:49:02Z

-	if sc.IdentityInstanceName != "" {
-		if _, ok := bc.CertProviderConfigs()[sc.IdentityInstanceName]; !ok {
-			return fmt.Errorf("identity certificate provider instance name %q missing in bootstrap configuration", sc.IdentityInstanceName)
-		}
-	}
-	if sc.RootInstanceName != "" {
-		if _, ok := bc.CertProviderConfigs()[sc.RootInstanceName]; !ok {
-			return fmt.Errorf("root certificate provider instance name %q missing in bootstrap configuration", sc.RootInstanceName)
-		}
-	}


For my education, why were these security checks deleted? Would it be users could now add their own generic security configs?

these files are just copied from internal xdsclient implementation to testutils of external xds client for testing. We don't have to do this kind of testing for external xdsclient so after copy, I have removed it. The only testing required for external client is resource watching scenarios.

danielzhaotongliu · 2025-03-08T00:12:40Z

 	// after that, and hence need not be guarded by a mutex.
-	transport          transport.Transport           // Takes ownership of this transport (used to make streaming calls).
-	ads                *ads.StreamImpl               // An ADS stream to the management server.
-	lrs                *lrs.StreamImpl               // An LRS stream to the management server.


To confirm for my understanding, this is because we are moving the LRS client functionality away from the xDS client into its own separate LRS client? (related to #8042).

Currently, in the immediate-term, we have no need for LRS, but we would need it in the future. So this is still a very nice to have.

yes, that's correct. For external client, LRS will have a separate client.

dfawley

Looks good, thanks! Please wait for @easwars to do a pass, too.

easwars · 2025-03-14T23:58:25Z

+func newXDSChannel(opts xdsChannelOpts) (*xdsChannel, error) {
+	switch {
+	case opts.transport == nil:
+		return nil, errors.New("xdsChannel: transport is nil")


We need to think about the error prefix used in these errors.

does xdsChannel make any sense to the user?

how does this error appear to the user? what context does the caller of this function add to the error.

do these error strings provide any actionable feedback to the user?

Earlier when this was all part of the internal xds client, these things did not matter so much. But now, if we are propagating errors to the users, they should be meaningful and actionable.

I added xdsclient as the prefix because that's the top level package name. I think message is straightforward?

If the user sees an error message xdsChannel: transport is nil, how will they go about handling it. This function newXDSChannel is called by internal code anyways. What I'm trying to say that if this is an internal API which is only called by code that we control, we should either ensure that it is always called with all parameters filled in, or return errors that are meaningful to the user and actionable as well.

if this is an internal API which is only called by code that we control, we should either ensure that it is always called with all parameters filled in, or return errors that are meaningful to the user and actionable as well.

Yeah so in the next PR when we implement the top level xDS client constructor, we can ensure that and return actionable error message

…ypeGetter

easwars · 2025-03-17T18:09:53Z

-// This is kept in internal until the gRPC project decides whether or not to
+// This is kept in internal until the clients project decides whether or not to
 // allow alternative backoff strategies.
 package backoff


If DefaultExponential is the only thing that is being used by the code, that should be the only thing exported. Everything else can be unexported.

easwars · 2025-03-17T18:12:20Z

Doug and I ran into this file (in our current codebase) as part of something else, and thought it was good idea to not do this anymore. So, please remove this file and just do a net.Listen("tcp", "localhost:0") wherever you want to start a new listener.

easwars · 2025-03-17T18:14:28Z

+// ResourceWatchStateForTesting returns the ResourceWatchState for the given
+// resource type and name.  This is intended for testing purposes only, to
+// inspect the internal state of the ADS stream.
+func (s *adsStreamImpl) ResourceWatchStateForTesting(typ ResourceType, resourceName string) (resourceWatchState, error) {


This is unused. Can it be removed?

easwars · 2025-03-17T18:18:09Z

+
+// processClientSideListener checks if the provided Listener proto meets
+// the expected criteria. If so, it returns a non-empty routeConfigName.
+func processClientSideListener(lis *v3listenerpb.Listener) (*listenerUpdate, error) {


There is very little detail in that bug. I personally wouldn't remember what I would need to remember based on that one-liner in a couple of weeks from now.

danielzhaotongliu · 2025-03-17T19:49:05Z

+	Multiplier float64
+	// Jitter is the factor with which backoffs are randomized.
+	Jitter float64


Would there be any validation for Multiplier and Jitter, it is unlikely, but could a user pass in a negative or zero value?

I have unexported them. Also, this package is internal only so user won't have access to them.

danielzhaotongliu · 2025-03-17T20:46:41Z

+			// not actually validating what the server sent and therefore don't
+			// know that it's invalid.  But we shouldn't ACK either, because we
+			// don't know that it is valid.
+			s.logger.Warningf("%v", nackErr)


nit: this should have a more detailed warning message regarding the server sending something the xDS client does not know about.

I think the nackErr will contain the reason as well since there can be multiple reasons?

danielzhaotongliu · 2025-03-17T20:51:31Z

+// If there are any errors decoding the resources, the metadata will indicate
+// that the update was NACKed, and the returned error will contain information
+// about all errors encountered by this function.
+func decodeResponse(opts *DecodeOptions, rType *ResourceType, resp response) (map[string]dataAndErrTuple, xdsresource.UpdateMetadata, error) {


Would this function be thread-safe? My understanding is that this would require synchronization by the caller?

yeah ResourceType.Decoder needs to be thread safe as well for the function to be thread safe

danielzhaotongliu · 2025-03-17T21:09:50Z

+		// Only the long form URL, with ://, is valid.
+		return &Name{ID: name}


nit: I think there should be a warning level log for this case. Ditto for similar if branches below where we just "silently fallback" and set the field ID to the input.

If the name isn't a valid new-style xDS name, field ID is set to the input. Note that this is not an error, because we still support the old-style resource names (those not starting with "xdstp:").

yeah as the docstring pointed its not an error. Besides, this is an internal helper function so may be the caller should ensure logging if required?

// The caller can tell if the parsing is successful by checking the returned Scheme.

…on (grpc#8144)

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch 2 times, most recently from cfedc55 to 957b0ae Compare March 5, 2025 15:50

purnesh42H added 4 commits March 6, 2025 09:43

copy common helpers for testing and logging

42faca1

common copied helpers modification

8a0900c

copy xdsclient helpers for logging and testing

36ffd55

xdsclient copied helpers modification

123a5a5

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from 957b0ae to 123a5a5 Compare March 6, 2025 04:13

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from a349898 to bf86ee7 Compare March 6, 2025 05:28

copy xdsclient channel implementation

03196ed

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from bf86ee7 to 03196ed Compare March 6, 2025 05:37

purnesh42H added Type: Feature New features or improvements in behavior Area: xDS Includes everything xDS related, including LB policies used with xDS. labels Mar 6, 2025

purnesh42H added this to the 1.72 Release milestone Mar 6, 2025

purnesh42H assigned dfawley and easwars Mar 6, 2025

purnesh42H requested review from dfawley and easwars March 6, 2025 08:53

xdsclient implementation modification

fba2d82

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from 4d7d9ae to fba2d82 Compare March 6, 2025 09:16

dfawley reviewed Mar 7, 2025

View reviewed changes

danielzhaotongliu reviewed Mar 8, 2025

View reviewed changes

purnesh42H added 3 commits March 10, 2025 01:02

remove unused code in common copied helpers

300c5a7

remove unused code in xdsclient helpers

e339ef4

dfawley suggestions #1

a1cf9cf

purnesh42H requested review from danielzhaotongliu and dfawley March 10, 2025 16:28

purnesh42H added 2 commits March 11, 2025 21:01

remove exported backoff package

33a2edb

remove xdsclient/internal/testutils completely

d78beb4

purnesh42H requested a review from dfawley March 12, 2025 14:56

purnesh42H assigned dfawley and unassigned purnesh42H Mar 12, 2025

rename listener_resource_type.go to helpers_test.go

f647a58

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from 538590e to f647a58 Compare March 12, 2025 15:05

dfawley approved these changes Mar 12, 2025

View reviewed changes

dfawley removed their assignment Mar 12, 2025

easwars reviewed Mar 15, 2025

View reviewed changes

easwars assigned purnesh42H and unassigned easwars Mar 15, 2025

easwars review 1: naming/nits

2b6c05c

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from b1fc3cb to 902e6d9 Compare March 16, 2025 20:45

easwars review 1: use clientConfig.ResourceTypes instead of resourceT…

1d347b6

…ypeGetter

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from 902e6d9 to 1d347b6 Compare March 16, 2025 20:45

copy testutils local listener, restartable listener and marshal

de4587f

purnesh42H requested a review from easwars March 17, 2025 16:16

purnesh42H assigned easwars and unassigned purnesh42H Mar 17, 2025

easwars approved these changes Mar 17, 2025

View reviewed changes

easwars assigned purnesh42H and unassigned easwars Mar 17, 2025

danielzhaotongliu reviewed Mar 17, 2025

View reviewed changes

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from f00537a to 5d6064c Compare March 18, 2025 05:13

easwar review 2

78fe33f

purnesh42H force-pushed the generic-xds-client-transport-channel-ads-stream branch from 5d6064c to 78fe33f Compare March 18, 2025 05:58

purnesh42H merged commit 1703656 into grpc:master Mar 18, 2025

purnesh42H mentioned this pull request Mar 19, 2025

xds: generic xds client resource watching e2e #8183

Merged

janardhanvissa pushed a commit to janardhanvissa/grpc-go that referenced this pull request Mar 24, 2025

xds: generic xDS client transport channel and ads stream implementati…

563cad1

…on (grpc#8144)

purnesh42H mentioned this pull request May 13, 2025

xds: Introduce new generic xDS and LRS clients decoupled from gRPC-specific elements and migrate internal xDS and LRS functionality to use new generic clients #8313

Closed

10 tasks

github-actions Bot locked as resolved and limited conversation to collaborators Sep 15, 2025

		ResourceTypeMapForTesting = make(map[string]any)
		ResourceTypeMapForTesting[xdsresource.V3ListenerURL] = listenerType

		// Only the long form URL, with ://, is valid.
		return &Name{ID: name}

Uh oh!

Conversation

purnesh42H commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

linux-foundation-easycla Bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

purnesh42H Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

purnesh42H commented Mar 5, 2025 •

edited

Loading

codecov Bot commented Mar 5, 2025 •

edited

Loading

linux-foundation-easycla Bot commented Mar 6, 2025 •

edited

Loading

purnesh42H Mar 10, 2025 •

edited

Loading