Using "message"s of imported proto files in rpc functions occurs "undefined" errors because functions are not generated in "pb" files

[farkow]

Missing functions when using messages from imported files in rpc functions

Using messages in imported proto files in rpc functions occurs undefined errors because functions are not generated in pb files.

Introduction

I have been doing some experiments with protobuf and jumped into a problem.
I have multiple proto files to generate a service in one single proto file.
After auto-generation, functions/messages from imported proto files are not in pb.go file.

I have already extracted the problematic part of my source here:
https://github.com/farkow/goprototest
It contains 2 different branches; master and working for you to check auto-generated files without any action.

Description

Let's say you will import a message from a different proto file into one other proto file to use it in an rpc call.

Imported proto file (image.proto - partially visible below)

...

message Image {
    string id = 1;
    string rev = 2;
    string note = 3;
    string file = 4;
}

message ImageCreateRequest {
    string api = 1;
    Image image = 2;
}

...

Main proto file (main.proto - partially visible below)

...

import "image.proto";
import "google/api/annotations.proto";

service Main {

    // image
    rpc CreateImage(image.ImageCreateRequest) returns (image.ImageCreateResponse) {
        option (google.api.http) = {
            post: "/v1/image"
            body: "*"
        };
    };

...

When you use the message in rpc call like in rpc CreateImage(image.ImageCreateRequest), auto-generated pb files are not generating those functions.
It exists in the interface, but the function is not generated.

// MainServer is the server API for Main service.
type MainServer interface {
	// image
	CreateImage(context.Context, *ImageCreateRequest) (*ImageCreateResponse, error)

However, when those definitions are in main proto file - just like in working.proto -, it can generate the functions.

func (x *ImageCreateRequest) Reset() {
	*x = ImageCreateRequest{}
	if protoimpl.UnsafeEnabled {
		mi := &file_working_proto_msgTypes[1]
		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
		ms.StoreMessageInfo(mi)
	}
}

There are 2 branches - master and working - for you to easily see differences in files under /pkg/api/v1 without running any command.

---

Prerequisites

• go
• protoc

Versions

• libprotoc 3.11.4
• go1.14.2 linux/amd64

What I did?

• I have imported image.proto and news.proto into main.proto
• I have created endpoints from the functions inside imported proto files
• I have generated pb.* files
  • Remove files inside /pkg/api/v1
  • Run make protoc-main for problematic version
  • Run make protoc-working for working version
• make build
  • For imported version (protoc-main), you will have errors
  • For combined version (protoc-working), it will successfully build

What I expect to see?

When I import other proto files, I expect to use their functions / messages in service block.

What I see instead?

undefined messages for all messages which comes from imported files inside service block.

Errors

After you run make protoc-main, main.pb* files will be generated under pkg/api/v1.
Run make build and you will have following errors:

pkg/api/v1/main.pb.go:130:39: undefined: ImageCreateRequest
pkg/api/v1/main.pb.go:130:86: undefined: ImageCreateResponse
pkg/api/v1/main.pb.go:131:37: undefined: ImageReadRequest
pkg/api/v1/main.pb.go:131:82: undefined: ImageReadResponse
pkg/api/v1/main.pb.go:133:36: undefined: NewsReadRequest
pkg/api/v1/main.pb.go:133:80: undefined: NewsReadResponse
pkg/api/v1/main.pb.go:134:36: undefined: NewsFindRequest
pkg/api/v1/main.pb.go:134:80: undefined: NewsFindResponse
pkg/api/v1/main.pb.go:145:59: undefined: ImageCreateRequest
pkg/api/v1/main.pb.go:145:106: undefined: ImageCreateResponse
pkg/api/v1/main.pb.go:145:106: too many errors

After you run make protoc-working, you will successfully build with make build.

[dsnet]

Fundamentally, the problem is that a go_package option needs to be specified with the full Go package path. protoc-gen-go has a bug that unfortunately suggested a poor value for certain use-cases. https://go-review.googlesource.com/c/protobuf/+/232338 fixes protoc-gen-go to stop doing that.

For the codebase you linked to, you'll want to change the go_package option in the following files:

• api/proto/v1/news.proto
• api/proto/v1/image.proto
• api/proto/v1/working.proto
• api/proto/v1/main.proto

to be:

option go_package = "github.com/farkow/goprototest/pkg/api/v1";

Closing, as the root cause is a poor go_package suggestion by protoc-gen-go, which is tracked by #1102.