Cannot install confluent-kafka with provided instructions on Apple's M1

[armenzg]

Description

I cannot install confluent-kafka on Apple's M1 host with neither Python 3.8 nor Python 3.9 with the provided instructions.

Workaround

This is a combination of what @evgmoskalenko posted here and what @emrahgunduz posted here:

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
# I believe this step needs to happen before exporting the flags
brew install openssl zstd pkg-config
# For compilers to find openssl@1.1 you may need to set:
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"
# From the librdkafka README, I believe we can skip this step since *just* configure is recommended
# ./configure --install-deps
./configure
make
sudo make install 

Once the above steps were taken I was able to install the Python package confluent-kafka on Python 3.8.10.

❯ python -c "from confluent_kafka import version, libversion; print(version()); print(libversion())"
('1.7.0', 17235968)
('1.8.0-RC1-4-gc4d569', 17301759)

How to reproduce

brew upgrade
brew install librdkafka
# I've also done tried this with Python 3.8.10
pyenv install 3.9.6 && pyenv local 3.9.6
python -m venv venv && source venv/bin/activate && pip install -U wheel pip && pip install confluent-kafka

Versions:

❯ brew -v
Homebrew 3.2.9
Homebrew/homebrew-core (git revision 7fcca92399; last commit 2021-08-24)
Homebrew/homebrew-cask (git revision fc93850e89; last commit 2021-08-24)
❯ pyenv -v
pyenv 2.0.4
❯ brew info librdkafka
librdkafka: stable 1.7.0 (bottled), HEAD
Apache Kafka C/C++ library
https://github.com/edenhill/librdkafka
/opt/homebrew/Cellar/librdkafka/1.7.0 (36 files, 4.8MB) *
  Poured from bottle on 2021-08-24 at 10:22:09
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/librdkafka.rb
License: BSD-2-Clause
==> Dependencies
Build: pkg-config ✔, python@3.9 ✔
Required: lz4 ✔, lzlib ✔, openssl@1.1 ✔, zstd ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 5,421 (30 days), 17,740 (90 days), 89,580 (365 days)
install-on-request: 3,440 (30 days), 11,254 (90 days), 55,667 (365 days)
build-error: 0 (30 days)

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

Version():                                    ('1.7.0', 17235968)
libversion():                                ('1.8.0-RC1-4-gc4d569', 17301759)
Apache Kafka broker version:   Not applicable
Client configuration:                  Not applicable
OS:                                              Apple Big Sur 11.5.2 - Apple M1
Provide client logs:                    Not applicable
Provide broker log excerpts:     Not applicable
Critical issue:                             It is not because there's a work around

Log:

❯ python -m venv venv && source venv/bin/activate && pip install -U wheel pip && pip install confluent-kafka
Collecting wheel
  Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.3)
Collecting pip
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-21.2.4 wheel-0.37.0
Collecting confluent-kafka
  Using cached confluent-kafka-1.7.0.tar.gz (103 kB)
Building wheels for collected packages: confluent-kafka
  Building wheel for confluent-kafka (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-wheel-xeoxwvu4
       cwd: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/
  Complete output (55 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.5-arm64-3.9
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
  copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
  copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/load.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.macosx-11.5-arm64-3.9
  creating build/temp.macosx-11.5-arm64-3.9/private
  creating build/temp.macosx-11.5-arm64-3.9/private/var
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/armenzg/code/M1/3_9/venv/include -I/Users/armenzg/.pyenv/versions/3.9.6/include/python3.9 -c /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c -o build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.o
  In file included from /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c:17:
  /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
  #include <librdkafka/rdkafka.h>
           ^~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
    Running setup.py install for confluent-kafka ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-record-jt0ajm2s/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/M1/3_9/venv/include/site/python3.9/confluent-kafka
         cwd: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/
    Complete output (55 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11.5-arm64-3.9
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
    copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
    copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/load.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    running build_ext
    building 'confluent_kafka.cimpl' extension
    creating build/temp.macosx-11.5-arm64-3.9
    creating build/temp.macosx-11.5-arm64-3.9/private
    creating build/temp.macosx-11.5-arm64-3.9/private/var
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/armenzg/code/M1/3_9/venv/include -I/Users/armenzg/.pyenv/versions/3.9.6/include/python3.9 -c /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c -o build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.o
    In file included from /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c:17:
    /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-record-jt0ajm2s/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/M1/3_9/venv/include/site/python3.9/confluent-kafka Check the logs for full command output.

[armenzg]

@jliunyu Hi! Would you mind providing some guidance on how to fix this? Or point me to someone that could give some pointers? Thanks!

[jliunyu]

@armenzg, I'm sorry for the late response. We didn't have a chance to test on Apple's M1 host yet.

@edenhill, do we have any plan for testing on Apple's M1 host?

[jliunyu]

Hi @armenzg , this looks like a path issue. Because librdkafka is installed with brew, but python doesn't look into the brew directories. So our suggestion is using the workaround you already used or find a way that python can find the brew installed packages.
From our side, we are looking into providing M1 binary wheels.

[armenzg]

Hi @jliunyu my apologies on the delay. I have no idea how I missed your reply.

I did some work on producing arm64 wheels in here. Cross-compiling does not work.

[armenzg]

Hi @jliunyu my apologies on the delay. I have no idea how I missed your reply.

I did some work on producing arm64 wheels in here. Cross-compiling does not work. You need an arm64 machine to produce it.