grpc-gateway calls golang/protobuf/jsonpb #178

mwitkow · 2016-05-13T08:45:50Z

This particular bug bit us:

To properly serialize/deserailize enums in jsonpb you need to call RegisterEnum. gogo/protobuf-generated messages register with gogo/protobuf/properties.go. However, if you have a library that is not under your control (e.g. gRPC Gateway), it will be using standard golang/jsonpb and thus the golang/protobuf/properties registration. Thus, any library using golang/jsonpb will not know about enums field maps.

Can we have a fix for this? Proposals include:

having gogo register by only with golang/protobuf
having gogo register with both, or passing it up.

The text was updated successfully, but these errors were encountered:

mwitkow · 2016-05-13T08:54:45Z

This is a bit of sed that helps:

  # import upstream golang/protobuf as well
  sed -i -E 's~import proto "github.com/gogo/protobuf/proto"~\0\nimport golang_proto "github.com/golang/protobuf/proto~' ${GO_FILE_FULL}
  # Double up all registrations to golang_protobuf
  sed -i -E 's~^([[:space:]]*)proto.Register(.*)$~\0\n\1golang_proto.Register\2~' ${GO_FILE_FULL}

awalterschulze · 2016-05-13T09:30:15Z

Damn that sucks :(
I'll have to think about this more.

I do have an extension that imports golang/protobuf instead of gogo/protobuf, maybe that can help?
This is what protoc-gen-gofast uses.

tamird · 2016-05-16T19:48:09Z

I'm not sure this is a "real" issue - truth is that using golang/protobuf/jsonpb to serialize/deserialize protos generated by gogo/protobuf doesn't work reliably (e.g. it doesn't support non-nullable non-scalar fields), so you really want to use gogo/protobuf/jsonpb.

The workaround isn't so bad, see my solution in cockroachdb/cockroach@2101119

awalterschulze · 2016-05-16T19:53:48Z

Ok maybe it just bothers me :-)
On 16 May 2016 9:48 PM, "Tamir Duberstein" [email protected] wrote:

I'm not sure this is a "real" issue - truth is that using
golang/protobuf/jsonpb to serialize/deserialize protos generated by
gogo/protobuf doesn't work reliably (e.g. it doesn't support non-nullable
non-scalar fields), so you really want to use gogo/protobuf/jsonpb.

The workaround isn't so bad, see my solution in cockroachdb/cockroach@
2101119
cockroachdb/cockroach@2101119

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#178 (comment)

mwitkow · 2016-05-16T20:19:15Z

Tamir, agreed. However if your vendored dependency used
golang/protobuf/jsonpb you're in very much of a pickle.

I think that having golang/protobuf generate MarshalJSON functions that use
jsonpb would be the best solution to the problem.

On Mon, 16 May 2016, 20:53 Walter Schulze, [email protected] wrote:

Ok maybe it just bothers me :-)
On 16 May 2016 9:48 PM, "Tamir Duberstein" [email protected]
wrote:

I'm not sure this is a "real" issue - truth is that using
golang/protobuf/jsonpb to serialize/deserialize protos generated by
gogo/protobuf doesn't work reliably (e.g. it doesn't support non-nullable
non-scalar fields), so you really want to use gogo/protobuf/jsonpb.

The workaround isn't so bad, see my solution in cockroachdb/cockroach@
2101119
<
cockroachdb/cockroach@2101119

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#178 (comment)

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#178 (comment)

awalterschulze · 2016-05-16T20:24:54Z

Sorry I replied to wrong issue
On 16 May 2016 9:53 PM, "Walter Schulze" [email protected] wrote:

Ok maybe it just bothers me :-)
On 16 May 2016 9:48 PM, "Tamir Duberstein" [email protected]
wrote:

I'm not sure this is a "real" issue - truth is that using
golang/protobuf/jsonpb to serialize/deserialize protos generated by
gogo/protobuf doesn't work reliably (e.g. it doesn't support non-nullable
non-scalar fields), so you really want to use gogo/protobuf/jsonpb.

The workaround isn't so bad, see my solution in cockroachdb/cockroach@
2101119
cockroachdb/cockroach@2101119

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#178 (comment)

awalterschulze · 2016-05-17T05:32:54Z

Line 58 of jsonpb_marshaler should probably be jsonpb.Unmarshal not
proto.Unmarshal
On 16 May 2016 21:48, "Tamir Duberstein" [email protected] wrote:

I'm not sure this is a "real" issue - truth is that using
golang/protobuf/jsonpb to serialize/deserialize protos generated by
gogo/protobuf doesn't work reliably (e.g. it doesn't support non-nullable
non-scalar fields), so you really want to use gogo/protobuf/jsonpb.

The workaround isn't so bad, see my solution in cockroachdb/cockroach@
2101119
cockroachdb/cockroach@2101119

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#178 (comment)

awalterschulze · 2016-05-17T08:04:51Z

I also think the generated MarshalJSON functions will solve all a lot of problems.

awalterschulze · 2016-09-10T08:26:55Z

Maybe we should make an issue with grpc-gateway to allow us to inject our own jsonpb marshaler codec, just like grpc-go allows you to inject a codec interface?

mwitkow · 2016-09-12T07:48:40Z

gRPC gateway already allows one to do that. It's just confusing as hell.

I actually think that double registration would be the better solution here.

awalterschulze · 2016-09-12T16:58:42Z

How is it confusing, maybe you can explain more? Worst case for someone else finding this issue. I haven't used grpc-gateway. I have only read about it.

mwitkow · 2016-09-12T19:11:55Z

For example: you use the default grpc gateway codes and gogo as your proto
compiler. Unknowingly you pass your messages to grpc gateway. Your enums
are not registered and serialised as ints and you can parse valid proto3 in
json format.

On Mon, 12 Sep 2016, 17:58 Walter Schulze, [email protected] wrote:

How is it confusing, maybe you can explain more? Worst case for someone
else finding this issue.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#178 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AJNWo6RtqaHe1CLWL2srs9l4-cnwY8LRks5qpYTEgaJpZM4Id0EI
.

awalterschulze · 2016-09-12T19:15:41Z

Ok so everything works in that case?

tamird · 2016-09-12T22:56:24Z

For reference: https://github.com/cockroachdb/cockroach/blob/995c4a0/pkg/util/protoutil/jsonpb_marshal.go

awalterschulze · 2016-09-13T05:58:01Z

Wow that is a lot of quite complex code, but as you said there are other issues with using golang/protobuf/jsonpb with gogoprotobuf generated structures, like non-nullable non-scalar fields. Have you brought this up with grpc-gateway? I would think that there would be an easier interface to implement.

tamird · 2016-09-17T12:28:34Z

I haven't, but it's probably worthwhile to do so.

Having to copy marshalNonProtoField is really unfortunate.

awalterschulze · 2016-09-19T17:17:01Z

Whoever opens this issue could CC me in on the issue. I would like to at least follow.

awalterschulze · 2016-10-18T15:45:20Z

Doesn't seem like the issue was ever opened #212 (comment)

johanbrandhorst · 2016-10-28T09:48:58Z

We're currently getting a proto: no coders for int message printed on STDERR when returning grpc.Codes over grpc-gateway, is that because of this? We vendored @tamird's workaround and everything else works, but are still seeing those messages. Could someone with a better understanding explain what's going on and preferably raise that issue?

awalterschulze · 2016-10-29T07:41:09Z

I would report this issue if I was using grpc-gateway, but unfortunately I am not. Also I don't perfectly understand. I am guessing. I would suggest putting in a panic at the place this warning it being printed out to find out how this call is happening. And then report this issue.

johanbrandhorst · 2016-10-31T12:25:03Z

Sounds reasonable, I'll give it a go

johanbrandhorst · 2016-11-01T11:29:53Z

For reference, I've managed to get a panic traceback, just posting it here before raising it against grpc-gateway:

2016/11/01 11:22:15 http2: panic serving 127.0.0.1:49722: proto: no coders for int
goroutine 427 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc4201381c0, 0xc4201e7f5f, 0xc420378b40)
    /usr/local/go/src/net/http/h2_bundle.go:4304 +0xd1
panic(0x8ac880, 0xc4202a6770)
    /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/proto.(*Properties).setEncAndDec(0xc42008ef20, 0xbfd8e0, 0x8ac140, 0xc4201e6eb0, 0x1)
    ~/myrepo/vendor/github.com/gogo/protobuf/proto/properties.go:381 +0x382
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/proto.(*Properties).init(0xc42008ef20, 0xbfd8e0, 0x8ac140, 0x88a710, 0x4, 0x88a720, 0x11, 0xc4201e6eb0, 0xc42008ef01)
    ~/myrepo/vendor/github.com/gogo/protobuf/proto/properties.go:713 +0xcb
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/proto.(*Properties).Init(0xc42008ef20, 0xbfd8e0, 0x8ac140, 0x88a710, 0x4, 0x88a720, 0x11, 0xc4201e6eb0)
    ~/myrepo/vendor/github.com/gogo/protobuf/proto/properties.go:699 +0x7f
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/jsonpb.jsonProperties(0x88a710, 0x4, 0x0, 0x0, 0xbfd8e0, 0x8ac140, 0x88a716, 0x28, 0x10, 0xc4202a65a8, ...)
    ~/myrepo/vendor/github.com/gogo/protobuf/jsonpb/jsonpb.go:892 +0xe1
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/jsonpb.(*Marshaler).marshalObject(0xc4202ccf80, 0xc4201e7518, 0x7f9012377670, 0xc4202ae2e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4202ec000)
    ~/myrepo/vendor/github.com/gogo/protobuf/jsonpb/jsonpb.go:222 +0x406
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/jsonpb.(*Marshaler).Marshal(0xc4202ccf80, 0xbeecc0, 0xc420308380, 0x7f9012377670, 0xc4202ae2e0, 0x1, 0xc420308380)
    ~/myrepo/vendor/github.com/gogo/protobuf/jsonpb/jsonpb.go:79 +0xca
github.com/johanbrandhorst/myrepo/vendor/github.com/gogo/protobuf/jsonpb.(*Marshaler).Marshal-fm(0xbeecc0, 0xc420308380, 0x7f9012377670, 0xc4202ae2e0, 0x8f9501, 0xc4202a62c0)
    ~/myrepo/vendor/github.com/cockroachdb/cockroach/pkg/util/httputil/http.go:71 +0x52
github.com/johanbrandhorst/myrepo/vendor/github.com/cockroachdb/cockroach/pkg/util/protoutil.(*JSONPb).marshal(0xc4202ccf80, 0x8e4580, 0xc4202ae2e0, 0x411a68, 0x20, 0x8f9540, 0x96a701, 0xc4202ae2e0)
    ~/myrepo/vendor/github.com/cockroachdb/cockroach/pkg/util/protoutil/jsonpb_marshal.go:57 +0x118
github.com/johanbrandhorst/myrepo/vendor/github.com/cockroachdb/cockroach/pkg/util/protoutil.(*JSONPb).Marshal(0xc4202ccf80, 0x8e4580, 0xc4202ae2e0, 0x35, 0x10, 0x8ed040, 0xbf75c0, 0xc4202ac720)
    ~/myrepo/vendor/github.com/cockroachdb/cockroach/pkg/util/protoutil/jsonpb_marshal.go:48 +0x3f
github.com/johanbrandhorst/myrepo/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime.DefaultHTTPError(0x7f90123337c0, 0xc4202ac720, 0xbf89a0, 0xc4202ccf80, 0xbf6b00, 0xc4201381c0, 0xc42037e960, 0xbef700, 0xc4202ae2c0)
    ~/myrepo/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/errors.go:91 +0x1cf
net/http.(*ServeMux).ServeHTTP(0xc420243680, 0xbf6b00, 0xc4201381c0, 0xc42037e960)
    /usr/local/go/src/net/http/server.go:2022 +0x7f
....
<my code>
....
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:4092 +0x6e2

johanbrandhorst · 2016-11-01T11:34:27Z

On second thought, I'd love to have some input from @tamird and @mwitkow about the precise cause of the problem here so I can make an educated issue against grpc-gateway. Any chance you could elaborate a little bit?

awalterschulze · 2016-11-01T12:37:17Z

Wow ok that trace is very interesting.
Could you also print out f *reflect.StructField

johanbrandhorst · 2016-11-01T12:49:36Z

With

panic(fmt.Sprintf("proto: no coders for Kind: %v, StructField: %v", t1, f))

2016/11/01 12:48:56 http2: panic serving 127.0.0.1:52008: proto: no coders for Kind: int, StructField: &{Code  int protobuf:"bytes,2,name=code" json:"code" 16 [1] fals
e}

awalterschulze · 2016-11-01T12:52:12Z

Where is it finding this int field?

johanbrandhorst · 2016-11-01T12:59:54Z

Using the debugger, I get this value for the valueField on ln 177 in gogo/protobuf/jsonpb/jsonpb.go:

valueField
<reflect.StructField>
Name:"Code"
PkgPath:""
Type:<reflect.Type>
Tag:"protobuf:"bytes,2,name=code" json:"code""
Offset:16
Index:<[]int> (length: 1, cap: 1)
Anonymous:false

The value of s is 386861805349424274, and i is 1

awalterschulze · 2016-11-01T13:45:37Z

Ok I'm stumped. Where is this Code field coming from?

johanbrandhorst · 2016-11-07T09:23:37Z

@tamird even without the use of gogo/protobuf generated files? I hadn't seen it in our logs before we made the switch but as I mentioned I haven't yet taken a closer look.

johanbrandhorst · 2016-11-11T15:00:48Z

I was able to reproduce the proto: no coders for int using the example provided by grpc-gateway, so I've raised grpc-ecosystem/grpc-gateway#259 now. I think this can probably be closed unless there's anything else to add?

awalterschulze · 2016-11-11T15:11:03Z

Good work

awalterschulze · 2016-11-20T11:50:54Z

I guess since grpc-ecosystem/grpc-gateway#259 is closed this issue can also be closed?

johanbrandhorst · 2016-11-21T09:06:03Z

At least the proto: no coders for int message is fixed now. I think the original issue is probably slightly different but I wouldn't object to closing for now anyway as the cockroachdb workaround seems to work well.

awalterschulze · 2016-11-21T19:19:50Z

@tamird This link is broken
For reference: https://github.com/cockroachdb/cockroach/blob/develop/util/jsonpb_marshal.go
Maybe someone still wants to bring an issue up with grpc-gateway about this interface?
Or maybe we could at least split out this code into a repo that others can use?

tamird · 2016-11-21T19:21:36Z

Updated the link.

johanbrandhorst · 2017-03-02T09:35:05Z

The underlying issue here has reared it's head again in enum parsing in the grpc-gateway. See grpc-ecosystem/grpc-gateway#320. Again it is because gogo/protobuf RegisterEnum doesn't register against golang/protobuf and so when grpc-gateway attempts to map an enum it fails to find it.

awalterschulze · 2017-03-02T09:49:21Z

Should we create a plugin to allow you to register with both gogoprotobuf and golang/protobuf or something like that.
I would personally like to create a plugin that does not import either, maybe these use cases can overlap?
Obviously not importing either will break extensions and ptypes.Any, but everything else should work if its implemented correctly.

johanbrandhorst · 2017-03-02T09:57:36Z

I would like something like that for better cross-compatibility.

awalterschulze · 2017-03-02T14:21:03Z

It would be useful for me (to try and to only build one new feature) if you could provide a repo which produces this problem, with an easy way for me to test.
Are you ok with no support for ptypes.Any and proto2 Extensions?

johanbrandhorst · 2017-03-02T14:27:58Z

@awalterschulze Pinging you on the gopher slack for further discussion

awalterschulze · 2017-03-02T14:31:05Z

Can we please keep the discussion here.

…

On Thu, 2 Mar 2017 at 15:28 Johan Brandhorst ***@***.***> wrote: @awalterschulze <https://github.com/awalterschulze> Pinging you on the gopher slack for further discussion — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#178 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvsLdvRKc-C8MqJsV24Ks2eCsyBT8FQks5rhtHvgaJpZM4Id0EI> .

johanbrandhorst · 2017-03-02T14:36:51Z

Okay, my org would have no problems with a plugin that doesn't support ptypes.Any or proto2 extensions. What kind of example are you interested in? The reason it's not working right now with the grpc-gateway is because it looks in a map created by golang/proto.RegisterEnum while all our generated files call gogo/proto.RegisterEnum

awalterschulze · 2017-03-02T14:52:14Z

Why does it look at this map? Is it only for json marshaling?

…

On Thu, 2 Mar 2017, 15:36 Johan Brandhorst, ***@***.***> wrote: Okay, my org would have no problems with a plugin that doesn't support ptypes.Any or proto2 extensions. What kind of example are you interested in? The reason it's not working right now with the grpc-gateway is because it looks in a map created by golang/proto.RegisterEnum while all our generated files call gogo/proto.RegisterEnum — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#178 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvsLS7onl0IEvvXczOzTMTYk1XiE1c9ks5rhtQDgaJpZM4Id0EI> .

johanbrandhorst · 2017-03-02T14:53:40Z

@awalterschulze It looks at the map so that it can map from the enum string to the enum code and vice versa. It's to allow things like https://my.com/api/v1/method?enum_variable=MY_ENUM_VALUE to be translated into the correct enum numeric representation.

johanbrandhorst · 2017-03-02T14:54:21Z

This is the relevant code: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/query.go#L150

johanbrandhorst · 2017-03-02T16:08:18Z

As a side note, if this is at all unpleasant to consider, @mwitkow sed commands can be run post-generation to fix this issue, so it doesn't need to implemented at this level.

awalterschulze · 2017-03-02T16:12:14Z

Ah ok, so grpc-gateway uses this in a totally different way. Thats unfortunate.
That means it has to register with github.com/golang/protobuf and there is no way around it :(

johanbrandhorst · 2017-03-02T16:13:57Z

Indeed

awalterschulze · 2017-03-02T16:19:53Z

Ok so you need a plugin that registers messages, enums, etc with both golang/protobuf and gogo/protobuf.
Maybe lets make a new issue for this?

johanbrandhorst · 2017-03-02T16:20:58Z

I'll get on it 😄

awalterschulze added the question label Aug 21, 2016

awalterschulze changed the title ~~gogo/protobuf should Register with upstream golang/protobuf~~ grpc-gateway calls golang/protobuf/jsonpb Sep 10, 2016

johanbrandhorst mentioned this issue Nov 11, 2016

"proto: no coders for int" printed whenever a gRPC error is returned over grpc-gateway. grpc-ecosystem/grpc-gateway#259

Closed

awalterschulze mentioned this issue Dec 10, 2016

NULL deserialization of Oneof when using GRPC #238

Closed

This was referenced Mar 2, 2017

Plugin for registering messages with golang/protobuf #269

Closed

Add dual registration plugin #270

Merged

awalterschulze closed this as completed in #270 Mar 6, 2017

johanbrandhorst mentioned this issue Feb 20, 2018

customtype: no encoder for custom.Uint128 #166

Closed

johanbrandhorst mentioned this issue Mar 28, 2018

runtime: support FieldMask as query param grpc-ecosystem/grpc-gateway#529

Merged

grpc-gateway calls golang/protobuf/jsonpb #178

grpc-gateway calls golang/protobuf/jsonpb #178

Comments

mwitkow commented May 13, 2016

mwitkow commented May 13, 2016

awalterschulze commented May 13, 2016

tamird commented May 16, 2016

awalterschulze commented May 16, 2016

mwitkow commented May 16, 2016

awalterschulze commented May 16, 2016

awalterschulze commented May 17, 2016

awalterschulze commented May 17, 2016

awalterschulze commented Sep 10, 2016

mwitkow commented Sep 12, 2016

awalterschulze commented Sep 12, 2016 • edited Loading

mwitkow commented Sep 12, 2016

awalterschulze commented Sep 12, 2016

tamird commented Sep 12, 2016 • edited Loading

awalterschulze commented Sep 13, 2016 • edited Loading

tamird commented Sep 17, 2016

awalterschulze commented Sep 19, 2016

awalterschulze commented Oct 18, 2016

johanbrandhorst commented Oct 28, 2016 • edited Loading

awalterschulze commented Oct 29, 2016

johanbrandhorst commented Oct 31, 2016

johanbrandhorst commented Nov 1, 2016

johanbrandhorst commented Nov 1, 2016

awalterschulze commented Nov 1, 2016

johanbrandhorst commented Nov 1, 2016 • edited Loading

awalterschulze commented Nov 1, 2016

johanbrandhorst commented Nov 1, 2016 • edited Loading

awalterschulze commented Nov 1, 2016 • edited Loading

johanbrandhorst commented Nov 7, 2016

johanbrandhorst commented Nov 11, 2016

awalterschulze commented Nov 11, 2016

awalterschulze commented Nov 20, 2016

johanbrandhorst commented Nov 21, 2016

awalterschulze commented Nov 21, 2016

tamird commented Nov 21, 2016

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017 via email

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017 via email

johanbrandhorst commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Mar 2, 2017

johanbrandhorst commented Mar 2, 2017

awalterschulze commented Sep 12, 2016 •

edited

Loading

tamird commented Sep 12, 2016 •

edited

Loading

awalterschulze commented Sep 13, 2016 •

edited

Loading

johanbrandhorst commented Oct 28, 2016 •

edited

Loading

johanbrandhorst commented Nov 1, 2016 •

edited

Loading

johanbrandhorst commented Nov 1, 2016 •

edited

Loading

awalterschulze commented Nov 1, 2016 •

edited

Loading