pickfirst: New pick first policy for dualstack #7498

arjan-bal · 2024-08-09T13:28:07Z

This PR moves the pickfirst logic into the LB policy as mentioned in A61.

Background

In Java and Go, pick first logic is implemented within the subchannel itself instead of inside the load balancer unlike C-core.

We will take this opportunity to bring gRPC to a more uniform architecture across implementations and write a new pick first policy. This is important so that different implementations do not continue to diverge as more features are implemented.

This change will include creating a PickFirstLeafLoadBalancer which will contain the pick first logic, as well as redesigning some components such as backoffs and address updates. This will set us up nicely to implement Happy Eyeballs and use pick first as a universal leaf policy.

The new pick first policy is not used by default and can be set using an environment variable. A GitHub actions workflow is created to run the tests with the env var set. Code coverage is also calculated by running with and without the env var set.

This implementation is based on the Java implementation.

RELEASE NOTES:

An experimental pick first LB policy is added which creates one subchannel per address. This is disabled by default until the Dualstack changes are completed.

codecov · 2024-08-09T13:31:55Z

Codecov Report

Attention: Patch coverage is 89.09091% with 30 lines in your changes missing coverage. Please review.

Project coverage is 82.15%. Comparing base (b8ee37d) to head (eb5a09a).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
balancer/pickfirst/pickfirstleaf/pickfirstleaf.go	89.01%	18 Missing and 12 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7498      +/-   ##
==========================================
+ Coverage   81.84%   82.15%   +0.30%     
==========================================
  Files         361      362       +1     
  Lines       27827    28102     +275     
==========================================
+ Hits        22775    23086     +311     
+ Misses       3850     3830      -20     
+ Partials     1202     1186      -16

Files with missing lines	Coverage Δ
balancer/pickfirst/pickfirst.go	`84.67% <100.00%> (+0.25%)`	⬆️
clientconn.go	`93.34% <ø> (+0.36%)`	⬆️
internal/envconfig/envconfig.go	`100.00% <ø> (ø)`
balancer/pickfirst/pickfirstleaf/pickfirstleaf.go	`89.01% <89.01%> (ø)`

... and 23 files with indirect coverage changes

zasweq · 2024-09-24T23:52:19Z

balancer/pickfirstleaf/pickfirstleaf.go

+	al.reset()
+}
+
+// seekTo returns false if the needle was not found and the current index was left unchanged.


Nit: please wrap all comments to 80 chars.

All of these nits applies to all comments. I also see different capital abbreviations in the comments (i.e. SubConn and subConn and others). Can you please standardize?

Wrapped the comments.
All the comments were using "subconn" while variable names were using "SubConn/subConn". Change the comments to also use "SubConn".

zasweq · 2024-09-24T23:58:05Z

balancer/pickfirstleaf/pickfirstleaf.go

+	b.subConns = resolver.NewAddressMap()
+}
+
+// deDupAddresses ensures that each address belongs to only one endpoint.


According to Mark, this is an implicit requirement out of the resolver (I asked about WRR and Outlier Detection). What should the behavior be in this undefined case? For OD, Mark mentioned he just uses the first endpoint that an address maps too, but the scenario where an endpoint wraps the same address as another isn't really a valid scenario.

I think that is in the context of EDS. Here, pick_first could be the top-level LB policy on the channel and there is no guarantee that the name resolver being used does not send duplicate addresses. Also, there is no complexity around localities here. The pick_first LB policy gets a flat list of endpoints or addresses.

Yes, there is no guarantee, but Mark mentioned in the xDS Chat duplicate addresses is an implicit bug in the resolver emissions. So behavior in OD/WRR is undefined in that scenario (mainly deployed in xDS Tree where EDS has a requirement of no deduping).

pick_first doesn't really care about endpoints, it will flatten the endpoint list and try to connect to all the addresses. The effect of for de-duplicating addresses is that we don't try to connect to the same address multiple times during a pass of pickfirst.
I've changed the implementation of UpdateClientConnState to flatten the endpoint list very early so that other parts of the LB policy handle a slice of addresses instead of a slice of endpoints. I initially started off converting the address list to endpoint list in UpdateClientConnState instead of the opposite after looking at Java's implementation, but its not really required.

zasweq · 2024-09-24T23:59:32Z

Can you please summarize what part of the gRFC this current PR implements in the PR description? (i.e. from a quick pass, I see the pick first iterations through the addresses one by one, but I don't see the happy eyeballs implementation)

…ly in UpdateClientConnState

arjan-bal · 2024-09-25T17:39:21Z

Can you please summarize what part of the gRFC this current PR implements in the PR description? (i.e. from a quick pass, I see the pick first iterations through the addresses one by one, but I don't see the happy eyeballs implementation)

This PR implements this section of A61: https://github.com/grpc/proposal/blob/master/A61-IPv4-IPv6-dualstack-backends.md#move-pick_first-logic-out-of-subchannel-javago

I've rephrased the first line of the PR description to emphasise what the hyperlink was pointing to.

dfawley

I wrote a couple comments against an older commit. Sending & starting over since they're getting lost behind later commits.

dfawley · 2024-09-24T17:50:49Z

balancer/pickfirstleaf/pickfirstleaf.go

+	// Can be changed in init() if this balancer is to be registered as the default
+	// pickfirst.


Nit: you should say something like "Is changed to pick_first in init() if this ..." so it's clear what it changes to, not just that it changes.

Updated the comment.

balancer/pickfirstleaf/pickfirstleaf.go

dfawley · 2024-09-30T21:34:24Z

.github/workflows/coverage.yml

@@ -19,6 +19,9 @@ jobs:
      - name: Run coverage
        run: go test -coverprofile=coverage.out -coverpkg=./... ./...

+      - name: Run coverage with new pickfirst
+        run: GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true go test -coverprofile=coverage_new_pickfirst.out -coverpkg=./... ./...


Does anything use this coverage_new_pickfirst.out file? How? By wildcard?

I verified that it gets used by checking that the coverage went up after adding this step.

Codecov docs just say that multiple reports are merged "automatically", but don't specify the logic:

https://docs.codecov.com/docs/github-4a-merging-reports

https://docs.codecov.com/docs/merging-reports

It is possible to explicitly specify the coverage files, we don't do that for the existing coverage.out filename.

dfawley · 2024-10-08T21:51:21Z

balancer/pickfirstleaf/pickfirstleaf.go

+
+func (b *pickfirstBalancer) newSCData(addr resolver.Address) (*scData, error) {
+	sd := &scData{
+		state: connectivity.Idle,


This is a bit intertwined with the behavior of the calling function -- it needs this to be Idle to cause it to connect. A comment here might be a good idea because of that.

Alternatively, we could make this call Connect (or make the caller always call Connect and do an early return) and set the state to Connecting.

The state stored in scData is updated only when the state notification is received by the stateListener. Since SubConns start in IDLE, I initialized the state with IDLE here.

grpc-go/clientconn.go

Lines 827 to 835 in b8ee37d

ac := &addrConn{

state: connectivity.Idle,

cc: cc,

addrs: copyAddresses(addrs),

scopts: opts,

dopts: cc.dopts,

channelz: channelz.RegisterSubChannel(cc.channelz, ""),

resetBackoff: make(chan struct{}),

}

Is there a reason to initialize the SubConn state with CONNECTING?

You would only initialize to CONNECTING if you also call connect.

Yeah I think this is all fine actually.

dfawley · 2024-10-08T21:52:02Z

balancer/pickfirstleaf/pickfirstleaf.go

+		// addresses within each endpoint. - A61
+		if cfg.ShuffleAddressList {
+			endpoints = append([]resolver.Endpoint{}, endpoints...)
+			internal.ShuffleAddressListForTesting.(func(int, func(int, int)))(len(endpoints), func(i, j int) { endpoints[i], endpoints[j] = endpoints[j], endpoints[i] })


I think this will need to be updated after your other PR.

Can we move this LB policy into the pickfirst package?

Moved the pickfirstleaf package inside the pickfirst package.

Is there any strong reason to keep it in a separate package at all? Why not put both PF policies inside the PF package? Longer term we will only have the one pickfirst again.

Also, in general, be very careful about adding exported packages and symbols if the plan isn't to keep them.

Marking the new policy as experimental is easier when the new policy is in its own subpackage. What functions/structs should be marked as experimental when I merge the packages?

Marking the new policy as experimental is easier when the new policy is in its own subpackage.

Is it? What is even exported from an LB policy package? Usually only the Name.

Discussed this offline. Sharing the same module would cause collisions in the unexported symbol names used by both packages such as logger, Name which. This would require renaming symbols and can cause cause confusion. Decided not to combine the packages.

dfawley · 2024-10-08T21:56:14Z

balancer/pickfirstleaf/test/pickfirstleaf_test.go

+ *
+ */
+
+package test


You can move these tests into pickfirstleaf and be package pickfirstleaf_test instead. That's nicer since they're colocated with the code they're testing, but they're still a separate package. Having a separate package for only tests is generally not ideal -- especially where the name of the package is just test, so it's hard to tell from the package name itself what is happening.

I agree, moved the files as suggested.

…ddress_per_subconn

arjan-bal added 18 commits August 8, 2024 23:04

Use one subconn per address in pickfirst

a1f262c

address review comments

9043079

ensure subConnList is never nil

2541d51

Defer subConnList creation till conection start

f961973

Move empty subConnList handling into refreshList function

cb5acc6

Fix race conditions

22e0bd4

Replace mutex with atomic for subConnList state

258145a

Use a go routine to manage phase 1 connection attempts

eeacec3

support both pickfirst together

632f4df

Use Java style implementation

69c6c7b

Revert comment change

67b3e93

Fix revive warnings

cceb75d

Synchronize using callback serializer

0d43e4b

Prepare for running all tests with new pf

b2a092a

Fix test failures

56bfb59

Enable test using new pf

17c63d2

Fix tests

f1daad1

Add unit tests

747bd8a

arjan-bal added this to the 1.66 Release milestone Aug 9, 2024

arjan-bal added the Type: Feature New features or improvements in behavior label Aug 9, 2024

arjan-bal requested a review from easwars August 9, 2024 13:28

arjan-bal assigned easwars Aug 9, 2024

Fix vet

84194db

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from f6a52fc to 84194db Compare August 9, 2024 13:38

Calculate coverage using new pickfirst also

586b091

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from d77dd20 to 586b091 Compare August 9, 2024 16:43

Fix lint error

31e8a10

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from e44b7a2 to 31e8a10 Compare August 9, 2024 17:31

simplify serializer usage

fcb0120

arjan-bal added 2 commits September 18, 2024 16:44

Fix lint

30256a5

Report TF on resolver error

6daf9cc

zasweq reviewed Sep 24, 2024

View reviewed changes

Comment wrapping, s/subcon/SubCon/g and flatten the endpoint list ear…

1f03ef9

…ly in UpdateClientConnState

dfawley reviewed Sep 25, 2024

View reviewed changes

balancer/pickfirstleaf/pickfirstleaf.go Outdated Show resolved Hide resolved

arjan-bal added 2 commits September 26, 2024 01:43

Don't close SubConns in resolverErrorLocked and fix comments

61c4816

Move common tests to pickfirst_test.go

2e9f873

arjan-bal requested a review from dfawley September 26, 2024 19:22

arjan-bal added 2 commits October 7, 2024 12:51

Set first pass when exiting idle

c4b4aa4

Ensure exitIdle doesn't increment the address list multiple times

f0e479e

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from 6ff5043 to f0e479e Compare October 7, 2024 08:49

Move handling of CONNECITN to IDLE subchannels to outer block

58e2ff7

dfawley reviewed Oct 7, 2024

View reviewed changes

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from 1ff881d to 0ee7d48 Compare October 8, 2024 09:33

Fix test log statement

3103efe

arjan-bal force-pushed the pickfirst_one_address_per_subconn branch from 0ee7d48 to 3103efe Compare October 8, 2024 16:00

dfawley reviewed Oct 8, 2024

View reviewed changes

arjan-bal added 2 commits October 9, 2024 15:16

Merge branch 'master' of github.com:grpc/grpc-go into pickfirst_one_a…

a40d619

…ddress_per_subconn

Organize packages based on review comments

e98bb80

dfawley approved these changes Oct 9, 2024

View reviewed changes

arjan-bal added 2 commits October 9, 2024 23:29

use onceFunc in idlePicker and reduce duplication when entering IDLE

a5b55e4

Fix comments

eb5a09a

arjan-bal merged commit 00b9e14 into grpc:master Oct 10, 2024
15 checks passed

arjan-bal deleted the pickfirst_one_address_per_subconn branch October 10, 2024 04:03

janardhanvissa pushed a commit to janardhanvissa/grpc-go that referenced this pull request Oct 15, 2024

pickfirst: New pick first policy for dualstack (grpc#7498)

23c4901

sagarsudo pushed a commit to sagarsudo/grpc-go that referenced this pull request Oct 16, 2024

pickfirst: New pick first policy for dualstack (grpc#7498)

4007b00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pickfirst: New pick first policy for dualstack #7498

pickfirst: New pick first policy for dualstack #7498

arjan-bal commented Aug 9, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

zasweq Sep 24, 2024

zasweq Sep 25, 2024 •

edited

Loading

arjan-bal Sep 25, 2024

zasweq Sep 24, 2024

easwars Sep 25, 2024

zasweq Sep 25, 2024

arjan-bal Sep 25, 2024

zasweq commented Sep 24, 2024 •

edited

Loading

arjan-bal commented Sep 25, 2024

dfawley left a comment

dfawley Sep 24, 2024

arjan-bal Sep 25, 2024

dfawley Sep 30, 2024

arjan-bal Oct 8, 2024

dfawley Oct 8, 2024

arjan-bal Oct 9, 2024

dfawley Oct 9, 2024

dfawley Oct 8, 2024

arjan-bal Oct 9, 2024

dfawley Oct 9, 2024

arjan-bal Oct 9, 2024

dfawley Oct 9, 2024

arjan-bal Oct 9, 2024 •

edited

Loading

dfawley Oct 8, 2024

arjan-bal Oct 9, 2024

		// Can be changed in init() if this balancer is to be registered as the default
		// pickfirst.

	ac := &addrConn{
	state: connectivity.Idle,
	cc: cc,
	addrs: copyAddresses(addrs),
	scopts: opts,
	dopts: cc.dopts,
	channelz: channelz.RegisterSubChannel(cc.channelz, ""),
	resetBackoff: make(chan struct{}),
	}

pickfirst: New pick first policy for dualstack #7498

pickfirst: New pick first policy for dualstack #7498

Conversation

arjan-bal commented Aug 9, 2024 • edited Loading

Background

codecov bot commented Aug 9, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

zasweq Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zasweq commented Sep 24, 2024 • edited Loading

arjan-bal commented Sep 25, 2024

dfawley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arjan-bal Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arjan-bal commented Aug 9, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

zasweq Sep 25, 2024 •

edited

Loading

zasweq commented Sep 24, 2024 •

edited

Loading

arjan-bal Oct 9, 2024 •

edited

Loading