tracking issue for upgrading protobuf to 3.19.0+ (and gRPC as well)

[nfelt]

Googlers, see also b/219030239.

This is a tracking issue for updating our protobuf dependency to 3.19.0 or greater. That would simplify our lives in several respects: it would solve #5703, and it would allow us to upgrade gRPC past 1.30.0 (blocked by grpc/grpc#23311).

That update would involve both updating the protoc compiler we use to generate Python bindings, as well as updating the Python protobuf runtime version we depend on (see below for what that means).

This is currently blocked on TensorFlow's own upgrade (which is blocked by tensorflow/tensorflow#53234; Googlers, see b/182876485). We're blocked on TF because a previous attempt to update protobuf to 3.18.1 resulted in build breakages for other TF-ecosystem projects that rely on us, since our generated protos were incompatible with the older protobuf runtimes required by those projects.

---

Explainer: the way we depend on protobuf is a bit subtle. We build our .proto files into python libraries (py_pb2 targets) using this tb_proto_library build rule: https://github.com/tensorflow/tensorboard/blob/master/tensorboard/defs/protos.bzl

(Example of tb_proto_library usage here)

The build rule depends on 2 components that are provided by the protobuf dependency:

• protoc - the protocol buffer compiler:

  tensorboard/tensorboard/defs/protos.bzl

  Line 35 in 454cfca

  protoc = "@com_google_protobuf//:protoc",

• the "protobuf Python runtime":

  tensorboard/tensorboard/defs/protos.bzl

  Line 28 in 454cfca

  runtime = "@com_google_protobuf//:protobuf_python"

The protoc compiler is the tool that at build time actually generates _pb2.py files from the .proto files. Note however that we check the generated pb2.py files into our pip package, so that users don't need to run protoc themselves to execute TensorBoard locally, but when building TensorBoard with Bazel we do run protoc for this reason (which has its own issues).

The protobuf Python runtime is a common library that provides much of the implementation of the generated _pb2.py files, and as such it is required to be present at runtime in order to actually use those proto messages. Also, the protobuf compatibility policy is that the runtime version must always be at least as high as the protoc version used to generated the _pb2.py files. The protobuf Python runtime provided as a Bazel build-time dependency when building with Bazel, which basically guarantees that we use the same version for both runtime and protoc. However, running the TensorBoard pip package, we don't have the Bazel-built dependencies available and instead get our dependencies from installed pip packages, where the protobuf runtime is provided by the pip package protobuf: https://pypi.org/project/protobuf/ That opens up the possibility of version skew, which is why we set a lower bound in our requirements.txt. This lower bound must always be at least as high as the protoc version used to compile the protos in that release of TensorBoard.

[saitcakmak]

As of protobuf 4.21.1 release, we're getting import errors while importing certain tensorboard components.

Repro:

conda create -n test_tensorboard python=3.7
conda activate test_tensorboard
pip install tensorboard
python
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

Trace:

>>> from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 24, in <module>
    from tensorboard.backend.event_processing import (
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 23, in <module>
    from tensorboard.backend.event_processing import event_file_loader
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 20, in <module>
    from tensorboard import data_compat
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/data_compat.py", line 20, in <module>
    from tensorboard.compat.proto import event_pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/compat/proto/event_pb2.py", line 17, in <module>
    from tensorboard.compat.proto import summary_pb2 as tensorboard_dot_compat_dot_proto_dot_summary__pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/compat/proto/summary_pb2.py", line 17, in <module>
    from tensorboard.compat.proto import tensor_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/compat/proto/tensor_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import resource_handle_pb2 as tensorboard_dot_compat_dot_proto_dot_resource__handle__pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/compat/proto/resource_handle_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import tensor_shape_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__shape__pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/tensorboard/compat/proto/tensor_shape_pb2.py", line 42, in <module>
    serialized_options=None, file=DESCRIPTOR),
  File "/Users/saitcakmak/opt/anaconda3/envs/test_tensorboard/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
>>> 

[lucasjinreal]

When wil lthis be merged?

[Thomas-MMJ]

any progress on this?

[nfelt]

This continues to be blocked by TensorFlow's issue, unfortunately.

[Thomas-MMJ]

So many machine learning projects depend on latest protobuf, but many others rely on tensorboard - could google create a 'protobuf' that is renamed something that doesn't conflict with the latest probobuf (say protobuf_legacy), then have pip, conda, etc. use the 'protobuf_legacy' for tensorboard till this is fixed?

[vadimkantorov]

should this be reopened for lifting <3.2.0 requirement?

[nfelt]

@vadimkantorov there isn't a <3.2.0 requirement, and certainly not from TensorBoard. You might mean the the <3.20 requirement from TensorFlow, which has already been lifted in this commit but hasn't yet been released in a stable release.

If you have a separate problem, please open a new issue.

[vadimkantorov]

Hmm, okay, I was confused by this recent behavior: pytorch/pytorch#90374, but maybe it occurred with an older version of tensorboard. If it actually works with newer protobufs it would be nice to note this explicitly in README, since this bug was around for a long time...

[nfelt]

The issue was fixed between when you filed that issue and now; the PR fixing it in January is linked earlier on this issue thread. We've released TensorBoard 2.12 which contains this fix, so if you have that version of TensorBoard installed you shouldn't see the PyTorch issue. We did mention the change in our release notes: https://github.com/tensorflow/tensorboard/releases/tag/2.12.0