reflect/protoreflect: add API to retrieve Descriptors

[jhump]

C++ and Java both provide a set of rich Descriptor types, that wrap the various *DescriptorProto messages defined in descriptor.proto. They provide a much easier-to-use API and do the heavy-lifting of working with descriptors: type/dependency resolution.

This enables, for example, a program to get a ServiceDescriptor, examine methods and corresponding request and response types (including the full request/response schema, including any nested messages from other imported files) and do something useful with them. One obvious example for using the descriptors would be an implementation of DynamicMessage (also provided in Java and C++ runtimes). (See #199)

Another clear use case is for dynamically interacting with a remote service that exports GRPC services that a given program does not know about (e.g. no need to generate from .proto files and re-compile -- instead interact with the service via reflection).

I've got a working example here:
https://github.com/jhump/protoreflect/tree/master/desc

As well as a use case, a client for GRPC service reflection:
https://github.com/jhump/protoreflect/tree/master/grpcreflect

[alandonovan]

Paraphrasing my comment on #295:

This is clearly useful functionality that a great many projects want, but my experience of two Google-internal packages providing this functionality (dynamicproto and descpool) suggests a piecemeal approach can easily result in a large, cumbersome, and concurrency-hostile API. We need a coherent design for the addition of descriptors and dynamic messages to the Go protobuf API, and there are several use cases that ought to be considered together:

• loading FileDescriptorProto messages from a variety of sources such as generated types linked into the application, .proto files in the file system, and external databases.
• converting the message representation of descriptors ("DescriptorProtos") into a cross-linked and indexed data structure ("Descriptors").
• instantiating messages of arbitrary types, not necessarily known at compile time, from a map[string]interface{} and a Descriptor, and the inverse operation of converting a message to a map.
• serializing/deserializing messages of arbitrary types to binary or text.

[jgiles]

It looks like #302 by @RJPercival is on pause pending this discussion? I was about to send an equivalent PR - I'm in a spot where I need the functionality in question (descriptor.ForEnum).

I fully agree that a larger plan on descriptors/dynamic messages in Go is a good idea. However, I'd argue that in the meantime, since we already have descriptor.Message + descriptor.ForMessage, and we already have EnumDescriptor() defined on all generated enum types, we should fill in the gap by adding a descriptor.Enum + descriptor.ForEnum. It's a pretty minimal addition that doesn't put us in much worse long term-shape than ForMessage already does if we decide to go in a different direction, but it unblocks folks who are content to work with the current setup until a better solution arrives.

[jhump]

FYI, @jgiles, I've now got a library that has seen a decent bit of production use and is fairly mature:
https://godoc.org/github.com/jhump/protoreflect

The main use cases are quite solid. There are some less-used (and less-thoroughly tested) regions of the dynamic message's API surface area where one may find some bugs (mostly interop/conversion between generated types and a dynamic message). But there are tests that end up indirectly exercising a lot of those code paths, too.

I invite you to take a look and provide feedback. It includes functionality analogous to #302 in the form of the desc.LoadEnumDescriptorForEnum(...) function.

[jgiles]

Thank you! I note the time on jhump/protoreflect@f3a731b - it's wonder when you ask for something on the internet and it appears 👍 (xooglers helping xooglers!). That addition solves my problem nicely.

The key problem I was trying to solve is the one you mention in that commit - there isn't a good way currently to go from an enum object (or the reflect.Type of one) to the fully-qualified proto enum type name. I still hope that some of this functionality makes it into the core Golang protobuf library.

[dsnet]

Putting on hold for #364, which fundamentally needs to have some representation of the proto type. I am unsure of whether it is going to be the Descriptor message itself or something else.

[dsnet]

https://golang.org/cl/127823 adds data structures for working with descriptors for the v2 API. The design is based on Go Reflection API for Protocol Buffers.

I'm going to marked this issue as fixed. Obviously, use of it will require the v2 API to become stable with a final release.

Note that the submitted API is not final, if you have concerns with it, feel free to file an issue.