No module named 'confluent_kafka.cimpl' AWS Lambda / Python 3.9

[temeke]

No module named 'confluent_kafka.cimpl'

How to reproduce:
Deploy AWS Lambda function with Python 3.9 runtime and try to import:

from confluent_kafka import Producer

When confluent_kafka version 1.8.2 is used it returns an error:
No module named 'confluent_kafka.cimpl'

Works with Python 3.7 and confluent_kafka version 1.7.0.

[jliunyu]

Hi @temeke, thanks for asking. Have you tried to install confluent_kafka with python 3.9?

This is command I used to install confluent_kafka on my local sudo python3.9 -m pip install confluent_kafka, please found the corresponding command to your system.

jliu@C02DM32YMD6T lib % sudo python3.9 -m pip install confluent_kafka 
WARNING: The directory '/Users/jliu/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting confluent_kafka
  Downloading confluent_kafka-1.8.2-cp39-cp39-macosx_10_9_x86_64.whl (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 4.8 MB/s eta 0:00:00
Installing collected packages: confluent_kafka
Successfully installed confluent_kafka-1.8.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
jliu@C02DM32YMD6T lib % python3.9
Python 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:09:00) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from confluent_kafka import Producer
>>> 

[iam-meghana]

Hi, I get the same issue with
from confluent_kafka import Producer

module = self._system_import(name, *args, **kwargs)
ImportError: dlopen(/Users/mv/venv/lib/python3.10/site-packages/confluent_kafka/cimpl.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_rd_kafka_AdminOptions_destroy'

I installed librdkafka using brew on macOS (Monterey 12.3 , Apple M1)
I'm using python 3.10.2

[edenhill]

@talktomeg Update your local librdkafka installation to the latest version (1.8.2)

[iam-meghana]

Hi @edenhill . It is the latest version.

brew info librdkafka

librdkafka: stable 1.8.2 (bottled), HEAD Apache Kafka C/C++ library https://github.com/edenhill/librdkafka /opt/homebrew/Cellar/librdkafka/1.8.2 (36 files, 4.9MB) * Poured from bottle on 2022-03-21 at 15:19:33 From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/librdkafka.rb License: BSD-2-Clause ==> Dependencies Build: pkg-config ✔, [email protected] ✘ Required: lz4 ✔, lzlib ✔, [email protected] ✔, zstd ✔

[edenhill]

Is that AdminOptions_destroy the only error you get?
If not, what's the full output?

You could try to specify the directory path to where librdkafka.dylib.1 resides with:
DYLD_LIBRARY_PATH=/some/path/to/homebrew/....../ python3 ./your_program.py ..

[iam-meghana]

Hi @edenhill ..Thank you for all the help!

The full error log is

Traceback (most recent call last):

  File "/Users/mv/examples/clients/cloud/python/./producer.py", line 25, in <module>
    from confluent_kafka import Producer, KafkaError

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/__init__.py", line 19, in <module>
    from .deserializing_consumer import DeserializingConsumer

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in <module>
    from confluent_kafka.cimpl import Consumer as _ConsumerImpl

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/cimpl.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_rd_kafka_AdminOptions_destroy'

However, I noticed that after rebuilding librdkafka from source and rerunning configure + make worked for me.

Appreciate the help! Thank you!

[Phillip14]

I'm having the same issue that @temeke mentioned originally in this issue. When running in an AWS Lambda using python 3.9 and confluent_kafka version 1.8.2 I get the following error: No module named 'confluent_kafka.cimpl'.

Has anyone found a solution to this?

[jliunyu]

I'm having the same issue that @temeke mentioned originally in this issue. When running in an AWS Lambda using python 3.9 and confluent_kafka version 1.8.2 I get the following error: No module named 'confluent_kafka.cimpl'.

Has anyone found a solution to this?

Hi @Phillip14 , how did you create the lambda function? For example, from a docker image, author from scratch.

[rbhatti999]

I'm also having the same issue as @Phillip14 and @temeke. However, I am using librdkafka 1.9.0 (so I am using confluent-kafka 1.9.0) and python 3.8 in PyCharm and python 3.8 as the my AWS Lambda Function Runtime. Getting exact same issue.

I created my lambda function code in PyCharm and uploaded to Lambda via .zip file and I uploaded the confluent-kafka library in a .zip file into a lambda layer.

If I have to use sudo python3.9 -m pip install confluent_kafka command do I need to uninstall my current installation of confluent-kafka?

[rbhatti999]

I found a solution that worked for me: https://stackoverflow.com/questions/63757476/error-while-using-confluent-kafka-python-library-with-aws-lambda

[Phillip14]

@jliunyu I'm authoring from scratch. Upon further review, I think the issue may lie in the way I'm building my lambda layer as the lambda is not able to import other packages as well (not just confluent_kafka). I will continue to debug from that perspective and look into the solution @rbhatti999 posted. Thanks for your responses!

[mhowlett]

triaging. marking as enhancement because we should do an AWS Lambda example, like the go client: confluentinc/confluent-kafka-go#823

[juliandecoss]

@temeke this resolves the problem, you should create a zip installing your dependencies with a image of a lambda from Docker and then add that to a zip ,then create a layer for your lambda, that works for me perfectly with Python 3.8 Lambda &Confluent

[sarth886]

Hi @edenhill ..Thank you for all the help!

The full error log is

Traceback (most recent call last):

  File "/Users/mv/examples/clients/cloud/python/./producer.py", line 25, in <module>
    from confluent_kafka import Producer, KafkaError

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/__init__.py", line 19, in <module>
    from .deserializing_consumer import DeserializingConsumer

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in <module>
    from confluent_kafka.cimpl import Consumer as _ConsumerImpl

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/confluent_kafka/cimpl.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_rd_kafka_AdminOptions_destroy'

However, I noticed that after rebuilding librdkafka from source and rerunning configure + make worked for me.

Appreciate the help! Thank you!

@says-megz Can you tell how can we build from source in mac

[sarth886]

librdkafka.dylib

@edenhill Any idea where the file will be located if we build from source