GoGo Protobuf looking for new ownership

[awalterschulze]

I write to you on behalf of the GoGo Protobuf project, which has long been supported by a small group of dedicated developers taking time out of their lives to keep the project up to date and add new features, to the benefit of all users. Unfortunately, the personal circumstances of the maintainers have changed and we are no longer able to keep up with the issues and feature requests that naturally crop up for a popular open source project.

In particular, the recent golang/protobuf release 1.4.x (AKA APIv2: https://blog.golang.org/protobuf-apiv2), has created a big chunk of work required to be compatible with the new world of Go protobufs. While longer term this work will likely make gogo/protobuf much more compatible with the rest of the ecosystem, there is currently no one to take on this piece of work, and users are living in uncertainty around the future of the project.

We are therefore reaching out to industry users such as yourself to ask for help with maintaining the project. Without external help, it is likely that the GoGo Protobuf project will be discontinued. We know there are many organisations out there that use the project, so we're hoping that one or more of them would step up to take on this maintenance.

Please let us know if you are a user that can take on (co)ownership of this project.
It is as simple as commenting on this issue.

Sincerely,

@awalterschulze and @johanbrandhorst

From Walter:

I want to thank @jmarais and @donaldgraham for doing a great job stepping up to take on maintenance of GoGo Protobuf over the past 2 years.
I want to also thank @johanbrandhorst for developing and maintaining the GRPC interoperability.
I am very lucky and proud to call you all my friends.

[alexanderbez]

I would really hate to see this project die and would love to see parity with the new Proto APIv2. I would like to help contribute as much as I can. Do you think before the existing core contributors move on, can create a milestone or set of issues needing to be completed to be compatible with the latest v2 updates?

[tac0turtle]

I share the same sentiment. I would like to contribute and help organize gogoproto. That being said I am unfamiliar with the inner workings of gogoproto and I am a junior developer, but I am willing to learn and work with others to help this project live on.

[johanbrandhorst]

The first task is to identity the current incompatibilities with APIv2. Just a quick look at the issue tracker gives me:

• WKT serialization is broken with golang/protobuf v1.4.0 and gogoproto_import = false #686
• gogo/proto marshalling is failing with latest github.com/golang/protobuf v1.4.0-rc.4 #678

There will probably be more, but it would be good to first get an idea of the exact shortcomings and necessary fixes. Assuming no bigger org steps up to take over, I think the short term goal is 1.4.x compatibility. If ownership can be transferred, there could be talk of a GoGo protobuf APIv2.

[dhiemaz]

Would love to take part of this project
@johanbrandhorst, @awalterschulze

[achille-roussel]

Hello @awalterschulze (and other gogo protobuf maintainers), thanks for opening up this discussion!

Would you be able to give a bit more details on how would this new ownership would be ideally structured?

• Are you trying to find a new home to step down as the leaders of this project?
• Are you looking to share the responsibility with a new group of maintainers?
• Would the current maintainers be interested in a sponsorship to spend some focus time on gogo protobuf?

[awalterschulze]

Thank you.
These are great questions.

Are you trying to find a new home to step down as the leaders of this project?

Yes and ideally a new home, where the employer(s) or team(s) acknowledges and gives those employee(s) at least some time to work on the project.

Are you looking to share the responsibility with a new group of maintainers?

No. All our circumstances have changed so much that this won't be possible.

Would the current maintainers be interested in a sponsorship to spend some focus time on gogo protobuf?

I know this is a No for me and Johan and highly probably a no for Donald and Jacques.

[ydnar]

We recently did some work to port a project from GoGo Protobuf to the v2 API. Aside from gogoproto.nullable (which we use quite a bit, in particular with WKTs), the blockers we identified were custom names, struct tags, and native well-known types.

Requests for these features are tracked here on the Go Protobuf repo:

• go_name: protoc-gen-go: support go_name option to specify Go identifier golang/protobuf#555
• Struct tags: protoc-gen-go: support go_tag option to specify custom struct tags golang/protobuf#52
• Standard Go types for well-known types: protoc-gen-go: support generating standard Go types in place of well-known-types golang/protobuf#414

The v2 reflection API might make implementing some of these easier than forking the project. If these features are important to you, I’d recommend weighing in on these issues.

[xen0n]

Reporter of #686 here. At work we're heavily reliant on gogo/protobuf for ergonomic gRPC APIs, and rewriting every enum and re-inventing a struct tag injector is not an option.

I tried to do some of the porting at that time, but it seems much of the machinery that the original maintainers used to track upstream is not public stuff (say, committed in this repo). On top of that it's generally hard to justify general open-source maintenance work when the department is under-staffed. We're watching this situation though, and I have just enough experience in compilers to review and debug code. If anyone steps up and decides to do the grunt work, I could probably spend a few weekends to help.

[tac0turtle]

Does someone want to compile a list of issues in a comment here & then we can work on creating a milestone to work towards?

It seems there are a few people that have stated interest in keeping this repo alive. And work towards upgrading this repo.

[johanbrandhorst]

Feel free to start with the ones linked already. We don't need a milestone, I don't think?

[johanbrandhorst]

#695 Is another one

[tac0turtle]

where is the best place to ask questions in regards to updating? is there a chat somewhere I can join or is here the best....?

[johanbrandhorst]

The #protobuf channel on Gophers slack is probably a good place to start.

[alexshtin]

I see many teams moving out of gogo. I considered the same for our project and run simple benchmark test to compared v1, v2, and gogo: https://github.com/alexshtin/proto-bench.

The results show that gogo is still ~2 times faster and v2 is actually slower than v1. We decided to stay on gogo. Unfortunately, we don't have resources to provide much help now but we might have them later.

[aaronc]

I would love to see this project maintained and would offer to contribute as I can. We currently are maintaining a fork https://github.com/regen-network/protobuf for use in https://github.com/cosmos/cosmos-sdk. I have a few open PR's that we are using actively on our fork: #657, #658, and #426

A few of my colleagues working on Cosmos/Tendermint. @alexanderbez and @marbar3778 have chimed in here as well so we may have some limited bandwidth, but we likely wouldn't want to be the only org responsible for maintenance and upgrading to v2. If anyone else is interested let us know.

It seems from the comments that there are parallel efforts to upgrade the v2 implementation to support some of the gogo proto features, yet that implementation is slower. Either way, it would be great to see the community align on a path forward rather than splintering - either upgrading gogo to v2 or porting gogo features to v2.

[awalterschulze]

• Struct tags: golang/protobuf#52

@ydnar after years of wishfully waiting for struct tags, it seems now that golang/protobuf has decided not to implement this feature :(

[ydnar]

• Struct tags: golang/protobuf#52

@ydnar after years of wishfully waiting for struct tags, it seems now that golang/protobuf has decided not to implement this feature :(

Agreed, I’m disappointed. We kind of stirred the pot with golang/protobuf#1142, but I’m glad that the Go protobuf maintainers have at least clarified their position.

We’ve prototyped an internal tool that wraps protoc-gen-go, intercepting the CodeGeneratorRequest and CodeGeneratorResponse, parses the Go source using the ast and types packages, and modifies the output before serialization.

I’ll share what we have when we have something working for (at least) our use cases.

[tandr]

@ydnar -- I think https://github.com/srikrsna/protoc-gen-gotag does exactly what you have described

[ydnar]

I’ll share what we have when we have something working for (at least) our use cases.

Progress update: we now have a working protoc-gen-go-patch tool that does what I described above. It uses the v2 protogen package and go/ast / go/types for most of the heavy lifting, and doesn’t require forking protoc-gen-go. I hope to extract it as a separate public package sometime next week.

Currently supported options: go_tags, go_name, go_message_name, go_enum_name, and go_value_name. TODOs include gRPC, oneof support, and a patch/gogo package for gogoproto.* options that alias to the above.

More interesting features like GoGo’s WKTs like time.Time are more complicated, but it’s a fun start.

[awalterschulze]

@ydnar this sounds very promising.
The TODO list as you describe should be highest priority.
I hope that customtype is not necessary, but do you think casttype would be possible?

gogoproto.* options that alias to the above.

This sounds like an interesting migration story.

[awalterschulze]

@ydnar could you give an example of how users specify the tags, names, etc?

Is it also using the protobuf extensions, like https://github.com/srikrsna/protoc-gen-gotag and gogoprotobuf or do you have a new method?

[ydnar]

@ydnar could you give an example of how users specify the tags, names, etc?

Is it also using the protobuf extensions, like https://github.com/srikrsna/protoc-gen-gotag and gogoprotobuf or do you have a new method?

Exactly. It uses extensions without a package name to allow proto files to specify short option names. Here’s an example:

https://gist.github.com/ydnar/fbb49f92412ecd6bb895b36e90c9a707

[ydnar]

Howdy, everyone. We’ve added a few more features to protopatch in the last few weeks, like support for embedded structs (thanks @Adphi!) and optional.

His most recent PR (alta/protopatch#61) will add support for overriding the type of a generated message field, ala Gogo’s casttype option.

[aaronc]

Maybe I should also mention https://github.com/cosmos/cosmos-proto. We have implemented fast marshaling and fast proto reflection for the golang proto v2 API. There are a few additional features in the works for our use case similar to customtype that may or may not be desirable for other protobuf users (cosmos/cosmos-proto#2 and cosmos/cosmos-proto#5)

[johanbrandhorst]

For yet another alternative to GoGo protobuf, see https://github.com/CrowdStrike/csproto. This was developed by Crowdstrike to fill the performance gap left between GoGo and the official Go protobuf implementation. It can also be used to migrate from APIv1 to APIv2 over time:

The runtime-agnostic APIs do the work of calling the correct runtime based on which generator was used to emit the Go struct so you don't run into problem when, for example, you pass a struct generated by protoc-gen-gogo to proto.Unmarshal() from google.golang.org/protobuf/proto. csproto.Unmarshal() inspects the underlying type of the msg parameter and will call Gogo's proto.Unmarshal() instead.

CC @dylan-bourque, the author.

[awalterschulze]

gogoprotobuf is now officially deprecated, so I'll be closed this issue.

I gave a final farewell to the project at GopherCon UK 2022, which you can watch here:
https://twitter.com/awalterschulze/status/1584553056100057088

Thanks again for all the support and I am sorry for any inconvenience caused.

[paralin]

If anyone is looking for a code-gen solution for Protobuf, I highly recommend vtprotobuf: https://github.com/planetscale/vtprotobuf

[dylan-bourque]

Related ... I created https://github.com/CrowdStrike/csproto if you need to support the V1 API (since vtprotobuf is API v2 only).

[cod-soap]

What a pity! I just want to use it in my project,
Thanks for you all great job

[alexshtin]

@cod-soap, you "just want" but we really did (3 years ago) and it is very deep integrated in our project. Moreover, I used one of those 40 strategies and created a soft fork for soft forked project. Thank you, @awalterschulze for the great job and great presentation. I am not complaining, just saying.

[awalterschulze]

@alexshtin Happy to hear you enjoyed the video :)

A soft fork of a soft fork wow, that must have been really tough. I am really sorry you had to go through that.

[Anagirotti]

• [ ]