go_proto_library dependency overhaul

[jayconrod]

Problem

A number of bugs have been reported recently about dependency conflicts when using go_proto_library, especially with gRPC: rules_go #1546, #1530, #1529, #1529, #1385, gazelle #219, #217, #183, #32

There are a couple root causes behind these bugs:

• go_proto_library rules may depend on go_libraries which were generated from .pb.go files. This sometimes works if the default Go plugin is used, if done consistently. However:
  • If an alternate protoc plugin is used, these files don't get regenerated.
  • We lose information about proto import paths, so the .pb.go files generated by the dependent go_proto_library may have the wrong imports.
  • If a go_proto_library for the same proto is included in the build, it creates compile-, link-, and run-time errors.
• We provide go_proto_library rules for the Well Known Types in @io_bazel_rules_go//proto/wkt, and Gazelle resolves imports to those rules. Our version of github.com/golang/protobuf, @com_github_golang_protobuf has special aliases for these rules. However, we don't have any special handling for other libraries in github.com/golang/protobuf.
  • Gazelle's special handling and the aliases make it hard to create a pure Go build without proto generation. This may make the build slower than necessary for people who just want to use .pb.go files, and may cause conflicts or unexpected versions if people have vendored these libraries.
  • There are go_libraries implicitly included by the go_grpc plugin. These cause conflicts.
  • There's no canonical proto_library or go_proto_library rules for gRPC or Google APIs. We should have a canonical set of rules for WKTs, gRPC, and Google APIs to avoid conflicts.

Basically, standard dependencies (Well Known Types, gRPC, and Google APIs) are a mess. Libraries are defined in several places (or not at all), and there's no clear, documented example showing how to use them.

Solution summary

• Declare @com_github_googleapis_googleapis in rules_go go_rules_dependencies(). Define proto_library rules for its contents.
• Provide go_proto_library and go_library rules for WKTs, gRPC, and Google APIs in @com_github_golang_protobuf and @org_golang_google_genproto. We have existing rules for some of these, and they will stay in their current locations.
• Remove alias rules in @com_github_golang_protobuf and @org_golang_google_genproto that point to @io_bazel_rules_go//proto/wkt. These were added since the last release and can be rolled back without (officially) breaking compatibility.
• Make Gazelle resolve Go and proto imports of WKTs, gRPC, and Google APIs to go_proto_library rules defined above.
• Add a disable_global proto mode to Gazelle to disable special handling of these imports when no go_proto_library dependencies are desired.

Use cases

Pure Go

Pure Go builds do not directly build anything from .proto files. However, they may incorporate go_libraries built from static .pb.go files, and they depend on common proto dependencies (github.com/golang/protobuf/proto, google.golang.org/grpc).
It may be necessary to build existing code this way when migrating to Bazel incrementally since Bazel can't build all .proto files as-is, especially vendored .proto files (vendor prefixes can't be trimmed from include paths in proto_libraries yet). In particular, Kubernetes needs to build this way, at least until bazelbuild/bazel#3867 is implemented.

For these builds, we should provide go_libraries for standard dependencies (WKTs, gRPC, Google APIs) without depending on go_proto_library or the proto compiler.

Pure proto

Developers should be able to generate Go code from .proto files as part of the build without the need to check in .pb.go files. This should work seamlessly for standard dependencies.

We must provide canonical proto_library and go_proto_library rules for all standard dependencies. Gazelle should know how to resolve imports in .proto and .go files to those dependencies.

Hybrid

The two cases above are easiest for support. However, large existing projects won't be able to migrate all at once, especially with dependencies using pure Go. Hybrid builds (with some go_proto_libraries and some static .pb.go files) should work reasonably well.

We may need to impose some restrictions:

• If any generated proto code is part of the build, when standard dependencies are imported, go_proto_library targets should be used, not go_library. go_proto_library with the default plugin adds standard WKT go_proto_library dependencies implicitly, and these will conflict with the equivalent go_libraries.
• If an alternate plugin is in use (e.g., gogo), it must generate code compatible with the static .pb.go files.
• .proto files that correspond to static .pb.go files cannot be imported. We won't be able to map these proto imports to Go imports for those, since we won't know what the .pb.go files were generated from.

Solutions

Rules for standard dependencies

We need to provide proto_library, go_proto_library, and go_library rules for the standard dependencies. The go_library rules should be used in pure Go builds. proto_library and go_proto_library rules should be used in pure proto and mixed builds.

proto_library rules are provided for Well Known Types in @com_google_protobuf (there are already BUILD files here; we don't generate them). There aren't standard proto_library rules for anything else. We should define a repository for github.com/googleapis/googleapis (there's no canonical name; following convention, let's call it @com_github_googleapis_googleapis), and provide proto_library rules using an overlay or a patch.

go_proto_library rules are provided for Well Known Types in @io_bazel_rules_go//proto/wkt. There are aliases for these rules in @com_github_golang_protobuf and @com_github_golang_genproto. It's unclear this is the right place for them. Ideally, go_proto_library rules should be defined alongside their proto_library rules, but we shouldn't rules to @com_google_protobuf. These rules could be defined alongside the static .pb.go files, but that would make it difficult for people to override those repositories. They will probably stay where they are.

We should provide go_proto_library rules for protos in @com_github_googleapis_googleapis. go_repository should be able to generate these correctly, so there's probably no need to check in custom build files.

go_library rules used to be generated for Well Known Types in @com_github_golang_protobuf and @com_github_golang_genproto. These rules were replaced with aliases to go_proto_library rules in @io_bazel_rules_go//proto/wkt, but that change is not released yet and may be rolled back. We should leave these are regular go_library rules generated from .pb.go files so that developers can easily override these repositories with go_repository.

@com_github_golang_genproto also contains go_library rules for gRPC and Google APIs, so no additional work needed there.

Gazelle and go_repository modes

Gazelle should support the use cases described above through modes that can be selected on the command line or through directives in build files.

For pure Go use cases, Gazelle will provide a disable_global proto mode, which will be opt-in. Developers can use disable_global mode by passing -proto=disable on the command line, by writing # gazelle:proto disable in the root build file, or by writing # gazelle:proto disable_global in any build file (only applies in that directory and subdirectories). The former two interfaces already exist; this change adds new meaning. Note that Gazelle currently enters disable mode automatically when visiting a vendor directory; this will not enter disable_global mode.

When in disable_global mode, Gazelle will resolve Go imports of known proto libraries (e.g., github.com/golang/protobuf/ptypes) as if they were regular Go libraries without any custom logic. If they exist in the local repository (e.g., in vendor), those libraries will be used. Gazelle will not generate, update, or delete existing proto_library or go_proto_library rules in pure mode (same as now).

When in the default proto mode or in non-global disable mode (implied by vendor directories), Gazelle may generate, update, and delete proto_library and go_proto_library rules. Gazelle will resolve proto and go imports of standard dependencies to canonical proto_libraries and go_proto_libraries, described above. For example, google/protobuf/any.proto will be resolved to @com_google_protobuf//:any_proto for proto, or @io_bazel_rules_go//proto/wkt:any_go_proto for Go. github.com/golang/protobuf/ptypes/any will also be resolved to @io_bazel_rules_go//proto/wkt:any_go_proto for Go. These special cases will be used instead of any libraries defined in the local repository with the same importpath.

The proto mode of the local repository doesn't affect how go_repository generates build files in external repositories, so in order to use disable_proto mode globally, users will need to set build_extra_args = ["-proto=disable"]. Even though this doesn't affect how repositories are fetched, we may want to promote this argument to a first-class attribute for visibility (like external or build_tags).

Impact to compatibility

• proto_libraries that import gRPC or Google APIs dependencies from a different location should continue to work. Gazelle will update these dependencies automatically. The location of the WKT proto_libraries is not changing.
• go_proto_libraries that import WKTs from @com_github_golang_protobuf// will now import go_libraries instead of go_proto_libraries. This was true in the last release; we're rolling back a flawed fix for this. Gazelle has resolved these dependencies to @io_bazel_rules_go//proto/wkt for a while already.
• go_proto_libraries that import gRPC or Google APIs dependencies from a different location may have conflicts (already true). Gazelle will update these automatically, so there may be some build file churn. The location of WKT go_proto_libraries is not changing.
• go_proto_libraries that import go_libraries for standard dependencies may have conflicts already. Gazelle will update these automatically, so there may be some build file churn.
• go_proto_libraries that import other go_libraries for proto dependencies may have conflicts already. Gazelle will fix these when go_proto_libraries are available.
  No changes are being made to the go_proto_library, go_library interfaces.
  Minimal changes are being made to Gazelle. If proto rule generation is already disabled globally, nothing needs to be done to opt into disable_global mode.

Documentation

We should extend proto/core.rst with instructions and examples on how to use and depend on gRPC and Google APIs. We should discuss the following (at least):

• Differences between pure Go, pure proto, and mixed builds, and why they're important.
• How to opt into pure Go builds.
• Locations of standard proto_library, go_proto_library, and go_library rules.
• Examples of how to use basic protobuf, protobuf with WKTs, basic, gRPC, and Google APIs.
• How to troubleshoot common issues
• Conflicts between libraries
• Can't import vendored proto libraries

Appendix: Proto repositories

Repository URL	Bazel name	Purpose
github.com/google/protobuf	@com_google_protobuf	Main proto compiler implementation. Has WKT .proto files. Defines proto_library rules in root package.
github.com/golang/protobuf	@com_github_golang_protobuf	Go proto plugin. Has WKT .proto and .pb.go files. We define proto_libraries here and aliases to go_proto_libraries defined in //proto/wkt.
google.golang.org/grpc a.k.a. github.com/grpc/grpc-go	@org_golang_google_grpc	gRPC runtime library, plus several more libraries for handling gRPC types. Does not contain protos or .pb.go.
google.golang.org/genproto a.k.a. github.com/google/go-genproto	@org_golang_google_genproto	Repository of generated .pb.go files (not .proto) for gRPC and Google APIs. When importing these libraries in Go, canonical paths point here.
github.com/googleapis/googleapis	none	Repository of .proto files for gRPC and Google Cloud APIs. Does not contain generated files. Has some Bazel files but no proto_library rules.

Appendix: Mapping between proto and Go import paths

Proto import	Go import	Note
google/*	github.com/googleapis/googleapis/google/*	gRPC, most other APIs
google/protobuf/*.proto	github.com/golang/protobuf/ptypes/*	Most WKTs
google/protobuf/api.proto, google/protobuf/field_mask.proto, google/protobuf/source_context.proto	google.golang.org/genproto/protobuf/*
google/protobuf/compiler_plugin.proto	github.com/golang/protobuf/protoc-gen-go/plugin
google/protobuf/descriptor.proto	github.com/golang/protobuf/protoc-gen-go/descriptor

[jayconrod]

In addition to supporting the standard plugin, we should also define go_proto_library rules and add support in Gazelle for gogoprotobuf. This support may improve over time, but this issue should not be closed without at least supporting the basic gogo_proto plugin.