Integration with other languages is partially broken

[doubleyou]

Looks like some (if not all) non-Go languages don't really work by default with protobuf files annotated with grpc-gateway bindings.

Consider the following gist: https://gist.github.com/doubleyou/41f0828e4b9b50a38f7db815feed0a6c

It basically implements HTTP bindings for protobuf's helloworld example. I also added the Makefile to give a bit better idea of what paths are being included, etc.

The problem arises when we try to run python server.py. Specifically, the code fails at the following line that gets generated inside helloworld_pb2.py:

from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2

The problem here is that we have never generated Python code from annotations.proto and http.proto and put them in the right place. I bet the same problem would arise with a lot of other languages, if not all of them.

This is an example of how people work around it right now: https://github.com/kasey/grpc-example/blob/master/installers/python-grpc-hack.sh. My guess is that in Ruby, C++ or Java world one has to do a very similar hack.

Seems like this is more of a Protocol Buffers problem: whenever you include an intermediate .proto file, you have to also generate code from it and put it to the appropriate include path. I'm not aware of any methods to make Protocol Buffers behave differently about it (for example, merge all the imported files into the main file and compile them as one?). Been trying to find any related issues raised in their repository or Google Groups, but no luck so far.

Another alternative is writing some helper scripts that compile annotations.proto and http.proto into a specified language and put them into the right place (just like in installers/python-grpc-hack.sh) and including them into the main repository. Can be a bit tricky, given that there are various ways of installing and running the same compiler/interpreter.

Finally, this can be at least documented for the time being, so that people know at least approximately where to look for a solution.

Thoughts, suggestions?

[doubleyou]

For what it's worth, it could have been as simple as adding a directory with the annotations_pb2.* and http_pb2.* modules into the project. However, due to the package name (https://github.com/grpc-ecosystem/grpc-gateway/blob/master/third_party/googleapis/google/api/annotations.proto#L17), the languages would expect the package google with a module named api. google pip package, however, already exists, so we can't just create a google/api directory in the project's directory, since it would basically override the installed google package, if it exists.

If inside the definitions if was something like this instead:

import "grpc-gateway/annotations";

then importing would have been trivial enough. I could see, however, this being a problem for the Golang ecosystem. Also, switching to the new layout would be a breaking change.

[achew22]

@doubleyou, Thanks so much for showing interest in the gRPC Gateway project. It's nice to know there are people out there playing with these ideas.

Unfortunately, I have almost no knowledge of the intricacies of the python import system and don't have any ideas on how to address your specific problem. Sorry. It sounds like this is a problem that would be better addressed by the Protobuf project. Maybe you could open a bug over there and add a reference to this one?

I'm going to close this issue since it is related to the actual protobuf compiler's python code generation strategy. Good luck out there and if you find a great solution, please come back and leave a note explaining how you solved the problem.

[doubleyou]

So, there's one way to make it work specifically for Python. If we generate the Python code from annotations.proto and http.proto, put them into a local nested google/api/ directory, and then add this to the local google/__init__.py:

import os
import pkgutil
stdlib_dir = os.path.dirname(os.__file__)
real_google_path = os.path.join(stdlib_dir, 'site-packages', 'google')
__path__.append(real_google_path)
execfile(os.path.join(real_google_path, '__init__.py'))

Then it will import both the native google package and the local one.

I'm still trying to search for any related discussions in the Protobuf community, but this may be at least a decent solution specifically for Python. Also, easy to automate.

[simjay]

Hi @doubleyou . Could you, if it is not so much a trouble, give me some example project that you could successfully run with grpc-gateway on Python? I have been trying to make this happen and tried all of your advices above but not getting anything working. Thanks a ton in advance.

[jackwootton]

I'm unsure why this was closed. I'm facing this exact issue with Python. The only GRPC example I can find doesn't work. Opened issue here GoogleCloudPlatform/python-docs-samples#1103

[navidnabavi]

Installing 'googleapis-common-protos' package from pypi will solve the problem.
pip install googleapis-common-protos

[dingrui37]

Hi @doubleyou could you give me a guide about how to use grpc-gateway in python? Thanks