TensorBoard's generated pb2.py files are incompatible with protobuf 4.21.0 #5703
Comments
|
Can you verify whether this works with a previous version of TensorBoard? It looks like you are using tensorboard==2.9.0. Can you try |
I only got the chance to verify that the issue was gone if I use tensorboard==2.9.0 and force protobuf to be 3.20.1 (by default it was 4.21.0rc1). According to https://developers.google.com/protocol-buffers/docs/news/2022-05-06, protobuf just updates from 3.20.1 to 4.21.0. |
|
The issue may be resolved if you install |
|
I saw the same error, and it appeared with both tensorboard 2.8.0 and 2.9.0 with protobuf 4.21.0rc1, but resolved when forcing protobuf to 3.20.1. |
This works. |
|
The more recent updates to protocolbuffers/protobuf#9954 seem to make it pretty clear that it's not actually grpcio-tools that fixes the issue, it's probably just that installing it forced a protobuf downgrade. The solution is probably that TensorBoard needs to either regenerate our |
This was referenced May 17, 2022
|
I've escalated to the protobuf folks at protocolbuffers/protobuf#9954 (comment). I also opened up a larger tracking issue for updating to 3.19.0+ in #5708. |
nfelt
changed the title
nfelt
changed the title
|
Originally posted by @saitcakmak in #5708 (comment) As of Repro: Trace: |
nfelt
added a commit
that referenced
this issue
Jan 19, 2023
Fixes #5708 and #5703. This updates our protobuf dependency to 3.19.6 in an attempt to address #5708, and provide a cleaner solution to #5703. The choice of 3.19.6 is meant to satisfy two competing constraints: - Current Python protobuf runtimes (the 4.x series) only support generated code from protoc versions 3.19.0+, as discussed in https://protobuf.dev/news/2022-05-06/. As a result, prior to this change, TensorBoard's pip package had to force its pip package dependency to `protobuf < 4` to avoid the errors seen in #5703. This PR lifts that restriction. - Current TensorFlow is still stuck on protobuf 3.x, the same as we have been, and as a result pins its pip package dependency using `protobuf < 3.20` (this could presumably be relaxed to `< 4` but that would require new TF releases). As a result, we must support at least one protobuf runtime version that also works with TF's constraints. Our previous attempt at this upgrade (to ~3.18 or so) caused test failures for Keras (which depends on TB, via TF, for the summary API code), apparently due to a protobuf runtime that was too old for our generated code. At the time, this was puzzling because they were pip-installing a protobuf runtime version that should have been recent enough - but I suspect now that this was a red herring, and bazel test was actually getting the protobuf runtime from the protobuf build dependency, not from the installed Python packages. If we see this failure mode again, we'll have to get Keras to update the protobuf Python runtime available in bazel tests. Lastly, this upgrade lets us clean up some additional issues we had to work around: - We can also upgrade gRPC now, to 1.48.2. I selected this version since it appears to be the most recent version prior to gRPC adopting protobuf 4.x (see grpc/grpc@41ec08c) - We can revert the backported fixes to protobuf and grpc from #5793, since the upgraded dependencies don't require patching - We can back out rules_apple reintroduction from #5561 that we only needed for gRPC
|
This should be resolved by #6147 which has been released in TensorBoard 2.12. If you are still seeing this error, please update to TensorBoard 2.12. |
|
I had to do this to solve the problem: |
yatbear
pushed a commit
to yatbear/tensorboard
that referenced
this issue
Mar 27, 2023
Fixes tensorflow#5708 and tensorflow#5703. This updates our protobuf dependency to 3.19.6 in an attempt to address tensorflow#5708, and provide a cleaner solution to tensorflow#5703. The choice of 3.19.6 is meant to satisfy two competing constraints: - Current Python protobuf runtimes (the 4.x series) only support generated code from protoc versions 3.19.0+, as discussed in https://protobuf.dev/news/2022-05-06/. As a result, prior to this change, TensorBoard's pip package had to force its pip package dependency to `protobuf < 4` to avoid the errors seen in tensorflow#5703. This PR lifts that restriction. - Current TensorFlow is still stuck on protobuf 3.x, the same as we have been, and as a result pins its pip package dependency using `protobuf < 3.20` (this could presumably be relaxed to `< 4` but that would require new TF releases). As a result, we must support at least one protobuf runtime version that also works with TF's constraints. Our previous attempt at this upgrade (to ~3.18 or so) caused test failures for Keras (which depends on TB, via TF, for the summary API code), apparently due to a protobuf runtime that was too old for our generated code. At the time, this was puzzling because they were pip-installing a protobuf runtime version that should have been recent enough - but I suspect now that this was a red herring, and bazel test was actually getting the protobuf runtime from the protobuf build dependency, not from the installed Python packages. If we see this failure mode again, we'll have to get Keras to update the protobuf Python runtime available in bazel tests. Lastly, this upgrade lets us clean up some additional issues we had to work around: - We can also upgrade gRPC now, to 1.48.2. I selected this version since it appears to be the most recent version prior to gRPC adopting protobuf 4.x (see grpc/grpc@41ec08c) - We can revert the backported fixes to protobuf and grpc from tensorflow#5793, since the upgraded dependencies don't require patching - We can back out rules_apple reintroduction from tensorflow#5561 that we only needed for gRPC
dna2github
pushed a commit
to dna2fork/tensorboard
that referenced
this issue
May 1, 2023
Fixes tensorflow#5708 and tensorflow#5703. This updates our protobuf dependency to 3.19.6 in an attempt to address tensorflow#5708, and provide a cleaner solution to tensorflow#5703. The choice of 3.19.6 is meant to satisfy two competing constraints: - Current Python protobuf runtimes (the 4.x series) only support generated code from protoc versions 3.19.0+, as discussed in https://protobuf.dev/news/2022-05-06/. As a result, prior to this change, TensorBoard's pip package had to force its pip package dependency to `protobuf < 4` to avoid the errors seen in tensorflow#5703. This PR lifts that restriction. - Current TensorFlow is still stuck on protobuf 3.x, the same as we have been, and as a result pins its pip package dependency using `protobuf < 3.20` (this could presumably be relaxed to `< 4` but that would require new TF releases). As a result, we must support at least one protobuf runtime version that also works with TF's constraints. Our previous attempt at this upgrade (to ~3.18 or so) caused test failures for Keras (which depends on TB, via TF, for the summary API code), apparently due to a protobuf runtime that was too old for our generated code. At the time, this was puzzling because they were pip-installing a protobuf runtime version that should have been recent enough - but I suspect now that this was a red herring, and bazel test was actually getting the protobuf runtime from the protobuf build dependency, not from the installed Python packages. If we see this failure mode again, we'll have to get Keras to update the protobuf Python runtime available in bazel tests. Lastly, this upgrade lets us clean up some additional issues we had to work around: - We can also upgrade gRPC now, to 1.48.2. I selected this version since it appears to be the most recent version prior to gRPC adopting protobuf 4.x (see grpc/grpc@41ec08c) - We can revert the backported fixes to protobuf and grpc from tensorflow#5793, since the upgraded dependencies don't require patching - We can back out rules_apple reintroduction from tensorflow#5561 that we only needed for gRPC
|
pip install -U protobuf==3.20.3 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Edited by @nfelt for context: The original title for this issue was
AttributeError: module 'google._upb._message' has no attribute 'Message'because that was the original error message with protobuf 4.21.0rc1. The final release protobuf 4.21.0 has a different error message,TypeError: Descriptors cannot not be created directly..Current status:
protobufto 3.19.0+ (and gRPC as well) #5708 tracks a more permanent fix to this problem.Our open source tests have recently encountered the following error which comes from tensorboard (or maybe protobuf):
Our pip install package list is as follows:
Can anyone help us investigate if this is a known bug? how do we unblock ourselves? Thanks!
The text was updated successfully, but these errors were encountered: