installation fails at mecab-python3==1.0.3

[abhinavdayal]

Describe the bug

Running pip install nemo_toolkit['all'] fails at installing mecab-python3==1.0.3 for python version 3.10.

As per mecab documentation 1.0.5 is python 3.10 compatible

Steps/Code to reproduce bug

On a system with python 3.10 installed do:
python3 -m venv venv
source ven/bin/activate
pip install Cython
pip install nemo_toolkit['all']

Expected behavior

It should install fine

Environment overview (please complete the following information)

• Ubuntu 22.04 LTS
• Method of NeMo install: tried with both pip and using git repo

Environment details

• Ubuntu 22.04 LTS
• Python version: 3.10.4

Additional context

Collecting mecab-python3==1.0.3

Using cached mecab-python3-1.0.3.tar.gz (77 kB)

Preparing metadata (setup.py) ... error

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.

│ exit code: 1

╰─> [44 lines of output]

  /hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.

    warnings.warn(

  WARNING: The wheel package is not available.

  WARNING: The wheel package is not available.

  running egg_info

  creating /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info

  writing /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/PKG-INFO

  writing dependency_links to /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/dependency_links.txt

  writing entry points to /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/entry_points.txt

  writing requirements to /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/requires.txt

  writing top-level names to /tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/top_level.txt

  writing manifest file '/tmp/pip-pip-egg-info-_e2t6_9n/mecab_python3.egg-info/SOURCES.txt'

  Traceback (most recent call last):

    File "<string>", line 2, in <module>

    File "<pip-setuptools-caller>", line 34, in <module>

    File "/tmp/pip-install-vrffzbza/mecab-python3_8c966bababd04383857aeb72d4c98956/setup.py", line 201, in <module>

      setup(name = "mecab-python3",

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup

      return distutils.core.setup(**attrs)

    File "/usr/lib/python3.10/distutils/core.py", line 148, in setup

      dist.run_commands()

    File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands

      self.run_command(cmd)

    File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command

      cmd_obj.run()

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 299, in run

      self.find_sources()

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 306, in find_sources

      mm.run()

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 541, in run

      self.add_defaults()

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults

      sdist.add_defaults(self)

    File "/usr/lib/python3.10/distutils/command/sdist.py", line 226, in add_defaults

      self._add_defaults_python()

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in _add_defaults_python

      self._add_data_files(self._safe_data_files(build_py))

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/sdist.py", line 132, in _add_data_files

      self.filelist.extend(

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 489, in extend

      self.files.extend(filter(self._safe_path, paths))

    File "/hdd/nemo/nemo-env/lib/python3.10/site-packages/setuptools/command/sdist.py", line 134, in <genexpr>

      for _, src_dir, _, filenames in data_files

  ValueError: not enough values to unpack (expected 4, got 1)

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: metadata-generation-failed

× Encountered error while generating package metadata.

╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

hint: See above for details.

[MaximumEntropy]

Is this exclusive to python 3.10? Does python 3.8 or 3.9 work for you?

[abhinavdayal]

I did not try on 3.9 yet. I tried building from source:

git clone https://github.com/NVIDIA/NeMo
cd NeMo
pip install -e .

Then when actually installing using reinstall.sh script, again got the same error.

[MaximumEntropy]

Okay, I could not reproduce this on Python 3.8.12 (inside the pytorch 22.03 container).

[abhinavdayal]

May I know where is the dependency mecab-python3==1.0.3 coming from? I don't see in any of the requirements.txt files. If I can change it to 1.0.5, it will go through as 1.0.3 is not compatible with python 3.10.

[abhinavdayal]

I installed Python 3.8, created venv and the installation worked fine.

[bermeitinger-b]

This shouldn't be closed. The error appears on Python 3.10.

[sqrk]

I have the same issue with Python 3.10

[shon-otmazgin]

also in python 3.9

[artbataev]

• For MacOS: brew install python && pip install cython && conda install -c conda-forge pynini => pip install 'nemo_toolkit['all']' works with Python 3.10, added to docs Add installation and ASR inference instructions for Mac #6377
• For Ubuntu: pip install cython => pip install 'nemo_toolkit['all']' works with Python 3.10. Checked with the new conda environment.

[olehrab]

Binary wheels are available for MacOS X, Linux, and Windows (64bit) are installed by default when you use pip:

pip install mecab-python3
These wheels include a copy of the MeCab library, but not a dictionary. In order to use MeCab you'll need to install a dictionary. unidic-lite is a good one to start with:

pip install unidic-lite
To build from source using pip,

pip install --no-binary :all: mecab-python3