Skip to content

Commit

Permalink
*: migrate grpc-gateway from v1.16.0 to v2.7.0
Browse files Browse the repository at this point in the history 
The PR is trying to migrate the api package from gogo to
google.golang.org/protobuf because the grpc-gateway runtime has been upgraded
to use protobuf-go v2 API. It's impossible to use gogo with new version
grpc-gateway and we can't re-generate all the proto codes by protobuf-go
because the raftpb uses gogoproto.nullable to use non-pointer for message
fileds, which protobuf-go doesn't support [1]. Fortunately, both
`api/etcdserverpb/rpc.proto` and
`server/etcdserver/api/v3electionpb/v3election.proto` doesn't use the
`gogoproto.nullable` features to message filed. For the migration, we can
just re-generate code for the following the protos as the approach:

* api/**/*.proto
* server/lease/leasepb/*.proto
* server/etcdserver/api/snap/snappb/*.proto
* server/etcdserver/api/v3election/v3electionpb/*.proto
* server/etcdserver/api/v3lock/v3lockpb/*.proto

Highlight the change for proto:

* Use `protoc-gen-go-vtproto` to generate Marshal/Unmarshal/Size code
  for `gogoproto.{marshaler,unmarshaler,size,}_all` interface, instead of
  using proto package;

* There is no way to support `gogoproto.goproto_enum_prefix_all`. So in
  this patch, I add `*.pb.ext.go` to add the const alias without prefix
  manually. Maybe we can build a proto binary as generator in the
  future;

* Both `api/etcdserverpb/etcdserver.proto` and `server/etcdserver/api/snap/snappb/snap.proto`
  use `gogoproto.nullable = true` for builtin type. In this patch, I upgrade
  the proto syntax from `2` to `3` with `optional` features. We can re-generate
  the optional fileds as pointer by `--experimental_allow_proto3_optional`.
  Since it's syntax upgrade, the marshal methods is also changed. For instance,
  in proto3 syntax, if the bool is empty, the marshaler won't encode it [2].
  It's compatible.

Highlight the change for grpc-gateway:

* The grpc-gateway/third_party/googleapis has been removed. We need to
  download the https://github.com/googleapis/googleapis.git to get the
  target proto.

* The protobuf-go won't generate json tag for `oneOf` filed [3]. We have
  to use `sed` in `scripts/genproto.sh` to append the `tag` before
  upstream accepts the change [4].

* Both `Hash` and `HashKV` shares the one URL `/v3/maintenance/hash`.
  New grpc-gateway openapi can detect the error. So in this patch, we
  should use `/v3/maintenance/hashkv` for `HashKV`.

* New grpc-gateway always renders default value in json payload. In this
  patch, we use set the `EmitUnpopulated: false`.

* The error message of grpc-gateway has been changed. In v1.16.0, the
  error message is rendered by [6]. In v2.7.0, the `error` string has
  been removed [7].

[1]: <golang/protobuf#1225>
[2]: <https://github.com/protocolbuffers/protobuf-go/blob/fc47fdd3d3fca5283fa9428ac94cf730236e4ca3/internal/impl/codec_gen.go#L74>
[3]: <https://github.com/protocolbuffers/protobuf-go/blob/fc47fdd3d3fca5283fa9428ac94cf730236e4ca3/cmd/protoc-gen-go/internal_gengo/main.go#L814>
[4]: <golang/protobuf#140>
[5]: <https://github.com/grpc-ecosystem/grpc-gateway/blob/40ee80cbcc0751e0c943d4abc99ca4bd196bec3b/docs/docs/development/grpc-gateway_v2_migration_guide.md?plain=1#L92>
[6]: <https://github.com/grpc-ecosystem/grpc-gateway/blob/v1.16.0/internal/errors.proto#L9>
[7]: <https://github.com/googleapis/go-genproto/blob/e85fd2cbaebc/googleapis/rpc/status/status.pb.go#L46>

Signed-off-by: Wei Fu <[email protected]>
  • Loading branch information
@fuweid
fuweid committed Jun 18, 2023
1 parent 6f2a5b7 commit 3b4e661
Show file tree
Hide file tree
Showing 87 changed files with 42,276 additions and 35,526 deletions.
187 changes: 103 additions & 84 deletions Documentation/dev-guide/apispec/swagger/rpc.swagger.json
View file

Large diffs are not rendered by default.

59 changes: 15 additions & 44 deletions Documentation/dev-guide/apispec/swagger/v3election.swagger.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
"title": "server/etcdserver/api/v3election/v3electionpb/v3election.proto",
"version": "version not set"
},
"tags": [
{
"name": "Election"
}
],
"consumes": [
"application/json"
],
Expand All @@ -25,7 +30,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -58,7 +63,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -91,7 +96,7 @@
"$ref": "#/definitions/v3electionpbLeaderResponse"
},
"error": {
"$ref": "#/definitions/runtimeStreamError"
"$ref": "#/definitions/rpcStatus"
}
},
"title": "Stream result of v3electionpbLeaderResponse"
Expand All @@ -100,7 +105,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -133,7 +138,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -166,7 +171,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -250,24 +255,15 @@
"protobufAny": {
"type": "object",
"properties": {
"type_url": {
"type": "string",
"description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n URL, or have them precompiled into a binary to avoid any\n lookup. Therefore, binary compatibility needs to be preserved\n on changes to types. (Use versioned type names to manage\n breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics."
},
"value": {
"type": "string",
"format": "byte",
"description": "Must be a valid serialized protocol buffer of the above specified type."
"@type": {
"type": "string"
}
},
"description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n Foo foo = ...;\n Any any;\n any.PackFrom(foo);\n ...\n if (any.UnpackTo(\u0026foo)) {\n ...\n }\n\nExample 2: Pack and unpack a message in Java.\n\n Foo foo = ...;\n Any any = Any.pack(foo);\n ...\n if (any.is(Foo.class)) {\n foo = any.unpack(Foo.class);\n }\n\n Example 3: Pack and unpack a message in Python.\n\n foo = Foo(...)\n any = Any()\n any.Pack(foo)\n ...\n if any.Is(Foo.DESCRIPTOR):\n any.Unpack(foo)\n ...\n\n Example 4: Pack and unpack a message in Go\n\n foo := \u0026pb.Foo{...}\n any, err := ptypes.MarshalAny(foo)\n ...\n foo := \u0026pb.Foo{}\n if err := ptypes.UnmarshalAny(any, foo); err != nil {\n ...\n }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n package google.profile;\n message Person {\n string first_name = 1;\n string last_name = 2;\n }\n\n {\n \"@type\": \"type.googleapis.com/google.profile.Person\",\n \"firstName\": \u003cstring\u003e,\n \"lastName\": \u003cstring\u003e\n }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n {\n \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n \"value\": \"1.212s\"\n }"
"additionalProperties": {}
},
"runtimeError": {
"rpcStatus": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"code": {
"type": "integer",
"format": "int32"
Expand All @@ -283,31 +279,6 @@
}
}
},
"runtimeStreamError": {
"type": "object",
"properties": {
"grpc_code": {
"type": "integer",
"format": "int32"
},
"http_code": {
"type": "integer",
"format": "int32"
},
"message": {
"type": "string"
},
"http_status": {
"type": "string"
},
"details": {
"type": "array",
"items": {
"$ref": "#/definitions/protobufAny"
}
}
}
},
"v3electionpbCampaignRequest": {
"type": "object",
"properties": {
Expand Down
26 changes: 11 additions & 15 deletions Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
"title": "server/etcdserver/api/v3lock/v3lockpb/v3lock.proto",
"version": "version not set"
},
"tags": [
{
"name": "Lock"
}
],
"consumes": [
"application/json"
],
Expand All @@ -25,7 +30,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -58,7 +63,7 @@
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/runtimeError"
"$ref": "#/definitions/rpcStatus"
}
}
},
Expand Down Expand Up @@ -107,24 +112,15 @@
"protobufAny": {
"type": "object",
"properties": {
"type_url": {
"type": "string",
"description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n URL, or have them precompiled into a binary to avoid any\n lookup. Therefore, binary compatibility needs to be preserved\n on changes to types. (Use versioned type names to manage\n breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics."
},
"value": {
"type": "string",
"format": "byte",
"description": "Must be a valid serialized protocol buffer of the above specified type."
"@type": {
"type": "string"
}
},
"description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n Foo foo = ...;\n Any any;\n any.PackFrom(foo);\n ...\n if (any.UnpackTo(\u0026foo)) {\n ...\n }\n\nExample 2: Pack and unpack a message in Java.\n\n Foo foo = ...;\n Any any = Any.pack(foo);\n ...\n if (any.is(Foo.class)) {\n foo = any.unpack(Foo.class);\n }\n\n Example 3: Pack and unpack a message in Python.\n\n foo = Foo(...)\n any = Any()\n any.Pack(foo)\n ...\n if any.Is(Foo.DESCRIPTOR):\n any.Unpack(foo)\n ...\n\n Example 4: Pack and unpack a message in Go\n\n foo := \u0026pb.Foo{...}\n any, err := ptypes.MarshalAny(foo)\n ...\n foo := \u0026pb.Foo{}\n if err := ptypes.UnmarshalAny(any, foo); err != nil {\n ...\n }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n package google.profile;\n message Person {\n string first_name = 1;\n string last_name = 2;\n }\n\n {\n \"@type\": \"type.googleapis.com/google.profile.Person\",\n \"firstName\": \u003cstring\u003e,\n \"lastName\": \u003cstring\u003e\n }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n {\n \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n \"value\": \"1.212s\"\n }"
"additionalProperties": {}
},
"runtimeError": {
"rpcStatus": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"code": {
"type": "integer",
"format": "int32"
Expand Down
7 changes: 7 additions & 0 deletions api/authpb/auth.pb.ext.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
package authpb

const (
READ Permission_Type = Permission_READ
WRITE Permission_Type = Permission_WRITE
READWRITE Permission_Type = Permission_READWRITE
)
Loading

0 comments on commit 3b4e661

Please sign in to comment.