proposal: protoc-gen-go: provide first-class support for plugins?

[dsnet]

There are number of issues filed in this repo that are related to meta-users asking for expanded and more stable support for plugins in protoc-gen-go via the generator.RegisterPlugin API.

For some background, the RegisterPlugin function and API was originally added to support gRPC. Given that was the original intended use-case, it was not well-thought out how it would be used by other projects in the open-source world.

The current amount of support for plugins is practically zero. The README states:

The protoc-gen-go/generator package exposes a plugin interface, which is used by the gRPC code generation. This interface is not supported and is subject to incompatible changes without notice.

Given the number of issues filed related to plugin support, it may be worth considering whether a stable API and documented restrictions should be provided to make plugins better supported.

I don't deny that there are benefits to the ecosystem of plugins that sprouted up. However, they also do carry serious maintenance burdens on the development of protoc-gen-go as it is easy for protoc-gen-go to make a change that breaks various plugin. For example, if we start generating an Unmarshal method, plugins that were previously generating that would suddenly break due to a duplicate method.

The alternatives to dropping plugin support are:

• Require users to fork the generator logic to bolt-on the logic themselves. However, this does make it difficult to compose the functionality of two plugins together.
• Treat plugins another post-processing step on the generated .pb.go file. They take the .pb.go file as input (and possibly the descriptor) and modify the input source or emit a separate file that exists side-by-side with that file. For example, a number of plugins exists that simply to add new methods could place the new methods in a separate file.

[ashi009]

I think the path forward should not focus on providing a way to extend protoc-gen-go, but on creating a framework for handling general protobuf code generation needs like importpath handling, type/field naming, and dependency management.

We can put plugins into two categories:

1. Modifying protoc-gen-go generated code (eg. adding additional tags to generated structure.)
2. Adding additional methods to the generated structures or exposing other structure from the proto package (eg. adding data verification methods.)

For the first type of plugins, a way to modify protoc-gen-go is required -- or wrap the protoc-gen-go and post-process the generated code (which sucks).

For the second type (which is also the common case), they can generate separate go files with the same package name with additional methods in it. Therefore they can run separately from protoc-gen-go.

Either way, a framework for writing go protoc plugin would be awesome.

[zellyn]

Plugins were instrumental in allowing us to use protobufs at Square to create our own Sake-alike and to add GRPC support. For instance, we had the ability to query service descriptors for years before GRPC added it as a supported function, and generated the requisite serialized descriptors using a plugin.

In the past, almost all suggestions, pull request, feature requests, etc. related to making plugins play more easily with each other were fobbed off with the response that it was an internal-only, unsupported thing.

So we wound up doing it anyway, except with far more pain. I believe it is a false dichotomy. It doesn't need to be be fully supported, with a backwards compatibility guarantee against breaking changes OR completely unsupported and completely immune to all improvements and making the interfaces more humane.

Why not have an as-decent-as-possible plugin architecture, with no stability guarantees? Right now we have a set of ugly forks, kludgy workarounds, and general hackiness, with no stability guarantee.

[neild]

The question to answer is what sorts of operations we can support in plugins.

I don't think we can support modification of the generated types in any meaningful fashion. Either the modifications we allow are so constrained as to be useless, or impossible to make any meaningful compatibility guarantee about. I'd be interested to hear specific counterexamples if there are any.

What we do (sort of) support is generating additional code. This actually doesn't need to be done as a protoc-gen-go plugin at all--you can write a program which runs as a protoc plugin (the same way protoc-gen-go does), parses the CodeGeneratorRequest, and produces whatever output it desires. Arguably this is what we should have done for grpc. Under this approach, the generated grpc (or whatever) service definitions end up in a different file (possibly in the same Go package, possibly not), but I don't see that as a negative.

[zellyn]

That is a good point. You can always generate more code to add functionality.

There are also places where support for interception/modification of existing functionality would be helpful, but I have work to do before submitting a particular proposal.

[zellyn]

Also, p.s. I see all the work and activity closing out the backlog of issues, and I deeply appreciate it.

[meling]

After having given this some thought, I'm cautiously optimistic that our relab/gorums plugin could be rewritten to work without running as a plugin, but as a standalone command that could generate our auxiliary code in a separate .gorums.go file that reside side-by-side with the .pb.go file. However, I would definitely want to have stable API support for getting hold of the various descriptors, so that we don't have to write a proto parser ourselves.

[dsnet]

Suppose there was a hypothetical package (let's call it protogen) that provided:

• Given a proto import path, protogen tells you the Go package path that the proto import path corresponds to (e.g., foo/bar/baz.proto maps to github.com/company/protos/foo/bar).
• Given a proto message name, protogen tells you Go type name for the message.
• Given a proto enum name, protogen tells you the Go type name for the enum.
• A documented guarantee that generated message types are of *T instead of T.
• A documented guarantee that generated enum types are of T instead of *T.

You could imagine that protogen also provides helpers to integrate with protoc and process the descriptor protos, but that's the icing on the cake. The functionality mentioned in the bullets above is critical for generators to cooperate.

Would you be able to implement gorums using protogen? If not, what's missing from protogen?

[meling]

I don't think the bullets would be enough for us. We currently use generator.FileDescriptor to get hold of not only message names, but also package name, method names, the request and reply message names, and options on those methods and more. We do that in order to populate a few data structures that is then used to generate Go code using a bunch of templates. This generated Go code uses gRPC and protobuf generated code. One of the key methods for extracting this information is here, and should be representative of the information we need.

[achew22]

What does the protobuf team think of protoc-gen-star? It seems to provide a lot of useful, standardized, functionality for protoc plugins.

[dsnet]

@achew22, the protobuf team operates out of github.com/protocolbuffers. This repository is maintained by a portion of the Go team for the Go implementation of protocol buffers.

That said, I'm not an expert on protoc-gen-star, but it seems that their goal is to make it easy to create protoc plugins across multiple languages. In particular, their support for Go is not always forwards compatible since they only rely on the CamelCase function to derive names. However, the purely lexicographical transformation that CamelCase does not take into account other information like gogo/protobuf's customname or a similar option that we may want to support officially (see #555). Furthermore, the current Go generator has some unfortunate logic for doing name conflict resolution that CamelCase alone cannot handle. Thus, I'm not sure I agree with the claim that it is "standardized" with regard to the Go ecosystem.

One of the goals for a hypothetical protogen package would be to provide a single place to make these naming decisions. If that package existed, then protoc-gen-star could theoretically use that to provide richer support for Go.

[dsnet]

The protogen package provides functionality for easily writing plugins for protoc, the protocol buffer compiler. We no longer support plugins to protoc-gen-go (which itself is a strange concept since protoc-gen-go itself is a plugin to protoc). The package provides the Go identifiers that are used to generate messages and enums so that you can write your own protoc plugins that generate Go code that depend on the generated Go code of protoc-gen-go.

It has been released as part of google.golang.org/[email protected].

[achew22]

What are the recommendations for protoc plugins that are written in Go, but do not emit golang? Is there a similarly supported package for that?

[neild]

You can either use protogen and ignore the Go-specific bits, or just write your own program. It's not hard; a protoc plugin is just a program that reads a CodeGeneratorRequest and writes a CodeGeneratorResponse.