Import Error in Python on M1 Mac

[BjornPrime]

What happened?

On Macs with M1 chips, urns may not build during installation. This problem has been traced to protobuf, so see protocolbuffers/protobuf#10417 for details.

Reporting here as well for tracking purposes.

Issue Priority

Priority: 2

Issue Component

Component: dependencies

[BjornPrime]

.take-issue

[Abacn]

My understanding to this issue is that Beam is using some internal APIs of protobuf and the distributed protobuf release may not contain them depending on platforms. Have to do a compile installlation of protobuf. First install the sources

brew install protobuf@3

For me this installs to /opt/homebrew/Cellar/protobuf@3/3.20.1

then set

export CPPFLAGS="-I/opt/homebrew/Cellar/protobuf@3/3.20.1/include"
export LDFLAGS="-L/opt/homebrew/Cellar/protobuf@3/3.20.1/lib"

then install protobuf with

pip install protobuf==3.20.1 --install-option="--cpp_implementation" --force

then python gen_protos.py should work.

This is only required when the user needs to compile beam from source. For the distributed beam releases the generated sources are included in release and the users do not need to do this.

[BjornPrime]

I can confirm that this does work for me. Will it work for any user attempting to compile from source or is access to this version of protobuf limited?

[BjornPrime]

I posted this issue in protobuf as well and they have closed in saying that it should be resolved in protobuf 4.21.x: protocolbuffers/protobuf#10417 (comment)

Unfortunately, grpcio-tools still requires protobuf<4.0, even in the most up-to-date current version (1.48.1).

[andreigurau]

Just wanted to chime in: I also have an M1 mac and struggled to get Beam working for Python. protobuf 3.20.x does not work for me (I get a KeyError: 'Unknown descriptor pool' error every time I run python gen_protos.py or a pytest command with those versions), but v3.19.4 works. I still have to run the above commands Abacn posted, but using ...protobuf/3.19.4/... instead of ...protobuf@3/3.20.1/...

That being said, after installing that pip installing that version, I'm seeing

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
googleapis-common-protos 1.57.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-cloud-vision 3.1.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-cloud-spanner 3.23.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-cloud-pubsub 2.13.11 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-cloud-dlp 3.9.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-cloud-bigquery 3.4.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.
google-api-core 2.10.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.4 which is incompatible.

(at the moment, that doesn't affect what I need to do, and I'm rarely contributing to the python SDK so it's not important right now, but in short, I'm not sure why I have to use a previous version of protobuf)

[tvalentyn]

Thanks for reporting, @andreigurau .
It would be good to find someone regularly working w/ Beam Python and has an M1, who is available to investigate further.

Also, I wonder how if we could repro this w/ github actions emulation of aarch64.

[BjornPrime]

I think @jrmccluskey might meet those criteria, Valentyn.

[jrmccluskey]

I have been beating my head into the wall with this one and I believe the protobuf issue was fixed on M1 for version 4.21.1 and greater, which is a big jump for our dependencies that we aren't ready for yet. I've been doing work SSH'd into a linux machine as a workaround for now

[tvalentyn]

which is a big jump for our dependencies that we aren't ready for yet.

Are you aware of what dependencies won't support protobuf 4 ? If so please comment on: #24569