etcd not compatible with grpc v1.30.0

[xiegeo]

To reproduce:

$ go get google.golang.org/grpc@latest
go: google.golang.org/grpc latest => v1.30.0
$ go mod tidy
...
go.etcd.io/etcd/v3/clientv3/naming imports
        google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.30.0), but does not contain package google.golang.org/grpc/naming

This is a problem for dependents of etcd who want to use the latest grpc.

The naming package was removed at grpc/grpc-go#3314 and was deprecated in favour of resolver.

I can work on a pull request if that's helpful.

[cfc4n]

I think rootcause is like #12068 , there is no go.mod in release-3.3 branch. go mod will download latest version of dependent package.

[ptabor]

As a workaround, you can pin it to 1.29.1:

require (
google.golang.org/grpc v1.29.1
)

But we should migrate out of the deprecated naming namespace.
(and in theory grpc should not deprecate within the same major version).

[junhwong]

go.mod
replace google.golang.org/grpc v1.30.0 => google.golang.org/grpc v1.26.0

[xiegeo]

I don't need a work around, I want to use grpc v1.30.0 to keep up to date.

[xiegeo]

After a bit of investigation. It looks like upgrading (switch from grpc/naming to grpc/resolver) isn't possible without breaking compatibility.

clientv3/naming has public function signatures that depend on grpc/naming. A migration, which needs to be complete eventually requires 2 steps:

1. Create a new package clientv3/resolver and depreciate clientv3/naming, this allows usage of grpc up to version 1.29.
2. Eventually, remove clientv3/naming, allowing usage of grpc v1.30 and up.

Alternatively, we can copy grpc/naming over to this project or its own namespace and rewrite imports. This still produces a breaking change but is much easier for users to adopt. Since grpc/naming should not be used anymore, switching to resolver is still the goal, but this allow keeping clientv3/naming and update grpc at the same time.

[xiegeo]

I have good news and bad news:

I did a little experimentation. Adding replace google.golang.org/grpc/naming => github.com/xiegeo/grpc-naming v1.29.1-alpha to go.mod file allows uploading grpc to v1.30.0 without any coding changes.

But another part of etcd is exposed as incompatible:

# go.etcd.io/etcd/v3/clientv3/balancer/picker
..\..\..\..\pkg\mod\go.etcd.io\etcd\[email protected]\clientv3\balancer\picker\err.go:25:9: cannot use &errPicker literal (type *errPicker) as type Picker in return argument:
        *errPicker does not implement Picker (wrong type for Pick method)
                have Pick(context.Context, balancer.PickInfo) (balancer.SubConn, func(balancer.DoneInfo), error)
                want Pick(balancer.PickInfo) (balancer.PickResult, error)
..\..\..\..\pkg\mod\go.etcd.io\etcd\[email protected]\clientv3\balancer\picker\roundrobin_balanced.go:33:9: cannot use &rrBalanced literal (type *rrBalanced) as type Picker in return argument:
        *rrBalanced does not implement Picker (wrong type for Pick method)
                have Pick(context.Context, balancer.PickInfo) (balancer.SubConn, func(balancer.DoneInfo), error)
                want Pick(balancer.PickInfo) (balancer.PickResult, error)

This is because of this change: grpc/grpc-go@4eb418e

The grpc maintiners went through the trouble of creating a V2Picker, then broke semantic versioning intentionally.

I think the best option is to bite the bullet and upgrade with the incompatible changes, we can do this while adding go.mod which already changes the module path.

[zerospiel]

@xiegeo Hi there, did you migrate proxy.grpcproxy package to your own grpc.resolver implementation? I've had some issues in getting SCs for proxy.grpcproxy.clusterProxy struct, so I just dropped the whole implementation.

In case you stacked with clientv3.balancer, let me know, it's fairy simple to migrate to new interfaces as of [email protected].

[xiegeo]

@zerospiel Nope, still waiting for the etcd core maintainers to decide what to keep and what to drop at each stage.

[nikhita]

We (Kubernetes) would also like to see etcd updated to use the latest grpc. Ideally, use [email protected] when it's released to include grpc/grpc-go#2628 (needed for kubernetes/kubernetes#93320).

@gyuho 👋 do you have suggestions on which of the above approaches to take here?

[zerospiel]

If anyone in a hurry to update to [email protected] and don't use service discovery using grpcproxy package — here is a fork https://github.com/ozonru/etcd/releases/tag/v3.3.0-rc.0-grpc1.30.0 where I just removed implementation for grpcproxy (due to undefined decision of how to implement it using grpc/resolver)

In go.mod: require github.com/ozonru/etcd/v3 v3.3.0-rc.0-grpc1.30.0

[pracucci]

I've attempted an upgrade in this draft PR #12155 but I'm having an issue due to the protobuf upgrade too. I didn't have much time to investigate it, but a second pair of 👀 would be very appreciated.

[nikhita]

@gyuho 👋 can you take a look?

[gyuho]

Hi,

Upgrading gRPC has always been painful where we had to basically rewrite client library. I will take a look to estimate how much work we need to upgrade gRPC and post updates here.

I am currently busy with other tasks.
In the meantime, did this workaround work? Does Kubernetes need new gRPC updates?

As a workaround, you can pin it to 1.29.1:

require (
google.golang.org/grpc v1.29.1
)

But we should migrate out of the deprecated naming namespace.
(and in theory grpc should not deprecate within the same major version).

/cc @jpbetz

[gyuho]

I am afraid if this requires us to rewrite our client third time just to make it compatible with gRPC :/

I will be working on some design doc about our long term plan. I would like to decouple clientv3 from gRPC as much as possible.

[dfawley]

I am afraid if this requires us to rewrite our client third time just to make it compatible with gRPC :/

I would recommend not using gRPC APIs explicitly marked as experimental in the documentation. This will avoid any potential future breakages. If you are missing functionality you need from our stable APIs, please reach out to us by filing an issue, and we will work with you to find a solution.

[gyuho]

@dfawley Thanks. Yes, going forward, we will rely less on the experimental APIs. Will definitely reach out to the gRPC team.

[sebito91]

Is it possible to update the etcd deps to support grpc using v1.31.0? There is a fix to the logger subsystem there that prevents extraneous logging by tools like loopyWriter:

grpc/grpc-go@506b773

[derekperkins]

Is there a timeline for the 3.5 milestone and whatever fix ends up getting committed?

[ramil600]

Currently as I see clientv3 still uses deprecated resolver:

func (c *Client) Dial(ep string) (*grpc.ClientConn, error) { creds := c.credentialsForEndpoint(ep) // Using ad-hoc created resolver, to guarantee only explicitly given // endpoint is used. return c.dial(creds, grpc.WithResolvers(resolver.New(ep))) }

If we try to update it with the new implemented resolver it will create import cycle, as I tried to do:

package go.etcd.io/etcd/client/v3 imports go.etcd.io/etcd/client/v3/naming/resolver imports go.etcd.io/etcd/client/v3: import cycle not allowed

What is the best way around for refactoring it? what do you think?

One solution I see is injecting []grpc.DialOption: *Client.Dial(ep string, dopts ...grpc.DialOption )

[bagualing]

I had the same problem today.

[ptabor]

etcd 3.5 uses grpc 1.41:

etcd/go.mod

Line 35 in b10adb6

google.golang.org/grpc v1.41.0

Please open a new issue with an exact description if there is another problem with:

• grpc version that is incompatible
• etcd version that has the problem

[prashantkumars289]

Which grpc version is compatible with etcd v3.3.27+incompatible.

If I am using latest grpc (v1.64.0), It's throwing error: it does not contain google.golang.org/grp/naming

For that I tried using grpc(v1.29.1). but getting following error on compilation

github.com/coreos/etcd/clientv3/balancer/resolver/endpoint

../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:87: undefined: resolver.BuildOption
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:40: undefined: resolver.ResolveNowOption

github.com/coreos/etcd/clientv3/balancer/picker

../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/err.go:37:53: undefined: balancer.PickOptions
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:63: undefined: balancer.PickOptions