ModuleNotFound Error: No module named 'lark'

[Sausebraus]

When building the example code lighting app for esp32s3 I get the above mentioned error. Even after running the following command: "source scripts/activate.sh && gn gen out/debug && ninja -C out/debug"

the output is:

[48/2051] ACTION //examples/all-cluste...en(//build/toolchain/host:mac_x64_gcc)
FAILED: gen/examples/all-clusters-app/all-clusters-common/app/PluginApplicationCallbacks.h gen/examples/all-clusters-app/all-clusters-common/app/callback-stub.cpp
python3 ../../third_party/pigweed/repo/pw_build/py/pw_build/python_runner.py --gn-root ../../ --current-path ../../examples/all-clusters-app/all-clusters-common --default-toolchain=//build/toolchain/host:mac_x64_gcc --current-toolchain=//build/toolchain/host:mac_x64_gcc --capture-output --python-dep-list-files gen/examples/all-clusters-app/all-clusters-common/all-clusters-common_codegen_codegen_metadata_path_list.txt -- ../../scripts/codegen.py --generator cpp-app --output-dir gen/examples/all-clusters-app/all-clusters-common --expected-outputs gen/examples/all-clusters-app/all-clusters-common/all-clusters-common_codegen.expected.outputs ../../examples/all-clusters-app/all-clusters-common/all-clusters-app.matter
Traceback (most recent call last):
File "/Users/fjprogramming/connectedhomeip/out/debug/../../scripts/codegen.py", line 28, in
from matter_idl.matter_idl_parser import CreateParser
File "/Users/fjprogramming/connectedhomeip/.environment/pigweed-venv/lib/python3.9/site-packages/matter_idl/matter_idl_parser.py", line 7, in
from lark import Lark
ModuleNotFoundError: No module named 'lark'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/fjprogramming/connectedhomeip/out/debug/../../scripts/codegen.py", line 32, in
from matter_idl.matter_idl_parser import CreateParser
File "/Users/fjprogramming/connectedhomeip/.environment/pigweed-venv/lib/python3.9/site-packages/matter_idl/matter_idl_parser.py", line 7, in
from lark import Lark
ModuleNotFoundError: No module named 'lark'
[57/2051] ACTION //examples/all-cluste...en(//build/toolchain/host:mac_x64_gcc)
ninja: build stopped: subcommand failed.

Does anybody have a solution for this error? I am working on an Intel MacBook.

[bzbarsky-apple]

@Sausebraus What's the last time you bootstrapped? It's possible that lark got added to the list of python modules we're pulling in during bootstrap after the last time you did that, in which case activating will get you that python environment but not the module.

So I would recommend running bootstrap and seeing if that fixes the problem.

[Sausebraus]

@bzbarsky-apple thanks for the quick response. The bootstrap actually fixed the problem in the terminal, yet I still get the same result in the demo project. Do I have to update something else in order for the demo project to receive the new files?

[bzbarsky-apple]

I'm not sure what you mean by "demo project"... Bootstrap should fix the problem in the terminal where it runs and any setting where "activate" is run after that bootstrap.

Depending on exactly what "demo project" is and how things actually run there, it may not be picking up the activate environment at all...

[Sausebraus]

the demo project is just the lighting app which is in the examples folder of matter. @bzbarsky-apple do you happen to know how things work there?

[bzbarsky-apple]

the demo project is just the lighting app which is in the examples folder of matter

OK, and what exact commands are you running to build it? It sounds like you are not building it from the terminal? How are you doing it?

[Sausebraus]

oh sorry @bzbarsky-apple , I am working in vs code with esp-idf installed. The microcontroller I am using is the ESP32-S3. I am building the project with esp-idf within vs code (command+shift+p and select build with esp idf).

Now I have reinstalled esp-idf with a guide from the Espressif homepage (select the correct board and under Installation > manual installation > Linux and macOS) and put into my .zprofile two commands:
export IDF_PATH=~/esp/esp-idf

and

export PKG_CONFIG_PATH="/usr/local/opt/openssl@3/lib/pkgconfig"

You can check if the IDF_PATH is correctly set by running printenv IDF_PATH or echo PKG_CONFIG_PATH. It should print out the path.

I don't know wether this has solved the issue or not because I am now facing new errors.

While building it prints out the following error:

[0/1] Re-running CMake...
-- Building ESP-IDF components for target esp32s3
Processing 3 dependencies:
[1/3] espressif/led_strip (1.1.0-alpha)
[2/3] espressif/mdns (1.0.7)
[3/3] idf (5.1.0)
-- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.10/bin/python3.10 (found version "3.10.9") found components: Interpreter
-- Project sdkconfig file /Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/sdkconfig
/Users/fjprogramming/.espressif/python_env/idf4.4_py3.9_env/bin/python: No module named kconfgen
CMake Error at /Users/fjprogramming/esp/esp-idf/tools/cmake/kconfig.cmake:176 (message):
Failed to run kconfgen
(/Users/fjprogramming/.espressif/python_env/idf4.4_py3.9_env/bin/python;-m;kconfgen;--list-separator=semicolon;--kconfig;/Users/fjprogramming/esp/esp-idf/Kconfig;--sdkconfig-rename;/Users/fjprogramming/esp/esp-idf/sdkconfig.rename;--config;/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/sdkconfig;--defaults;/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/sdkconfig.defaults;--env-file;/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/build/config.env).
Error 1
Call Stack (most recent call first):
/Users/fjprogramming/esp/esp-idf/tools/cmake/build.cmake:613 (__kconfig_generate_config)
/Users/fjprogramming/esp/esp-idf/tools/cmake/project.cmake:453 (idf_build_process)
CMakeLists.txt:32 (project)

-- Configuring incomplete, errors occurred!
See also "/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/build/CMakeFiles/CMakeOutput.log".
See also "/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/build/CMakeFiles/CMakeError.log".
FAILED: build.ninja
/Users/fjprogramming/.espressif/tools/cmake/3.23.1/CMake.app/Contents/bin/cmake --regenerate-during-build -S/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32 -B/Users/fjprogramming/connectedhomeip/examples/lighting-app/esp32/build
ninja: error: rebuilding 'build.ninja': subcommand failed

Did I mess up Ninja or CMake? I am so sorry if my description is unclear or dumb but I am very tired, I try to get this up and running for a few days now

[bzbarsky-apple]

1. If you updated esp-idf you probably need to manually delete the build dir. At least that's my experience with it... Otherwise it does tend to run inot ninja/cmake issues.
2. If you are building in vscode, it might not be running activate.sh... In which case it's not picking up the "right" python virtualenv. You might be able to just install lark in whatever python it is using, or maybe there's a way to get it to run activate.sh.

@shubhamdp or @dhrishi may know more about this.

[Tim4497]

Same error: downloaded esp-idf 5.0 new and reinstalled matter as well, botstrapping and activating, then moving to the examples folder, doing get_idf command and then trying to build with the following errors:

Traceback (most recent call last):
  File ".../Matter/connectedhomeip/scripts/codegen.py", line 28, in <module>
    from matter_idl.matter_idl_parser import CreateParser
ModuleNotFoundError: No module named 'matter_idl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../Matter/connectedhomeip/scripts/codegen.py", line 32, in <module>
    from matter_idl.matter_idl_parser import CreateParser
  File ".../Matter/connectedhomeip/scripts/py_matter_idl/matter_idl/matter_idl_parser.py", line 7, in <module>
    from lark import Lark
ModuleNotFoundError: No module named 'lark'

get_idf returns:

Checking "python3" ...
Python 3.9.5+chromium.19
python3 has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Not using an unsupported version of tool openocd-esp32 found in PATH: unknown.
Not using an unsupported version of tool ninja found in PATH: 1.11.1.
Checking if Python packages are up to date...
Constraint file: .../.espressif/espidf.constraints.v5.0.txt
Requirement files:
 - .../esp/esp-idf/tools/requirements/requirements.core.txt
Python being checked: .../.espressif/python_env/idf5.0_py3.9_env/bin/python
Python requirements are satisfied.
All paths are already set.

I did pip3 install lark which worked.

Had to do pip3 install stringcase aswell.

[tomasmcguinness]

I've found that some recent PRs have caused conflicts between macOS and Linux python setups.

On my Mac, I removed all the platform and linux arch values from the scripts/requirements.esp32.txt.

I was then able to run export.sh and activate.sh before compiling and flashing the all-clusters-app example.

The pull request #22071 wi will give you some insight. Seems there is some back and forth between macOS and linux environments.