Cannot Install tensorflow 2.9.1

[oscar-defelice]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS Monterey 12.5 - MacM1 processor
• Poetry version: 1.1.14
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/oscar-defelice/7606ebc658b8a6c8b7cf574c384a59dd

Issue

Here is the issue: I am not able to install tensorflow with poetry.
If I run poetry add tensorflow I get the following error

uilding wheels for collected packages: llvmlite
    Building wheel for llvmlite (setup.py): started
    Building wheel for llvmlite (setup.py): finished with status 'error'
    error: subprocess-exited-with-error
    
    × python setup.py bdist_wheel did not run successfully.
    │ exit code: 1
    ╰─> [11 lines of output]
        running bdist_wheel
        /Users/oscardefelice/miniconda3/envs/tempr/bin/python /private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py
        LLVM version... Traceback (most recent call last):
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 220, in <module>
            main()
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 214, in main
            main_posix('osx', '.dylib')
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 134, in main_posix
            raise RuntimeError(msg) from None
        RuntimeError: Could not find a `llvm-config` binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help.
        error: command '/Users/oscardefelice/miniconda3/envs/tempr/bin/python' failed with exit code 1
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for llvmlite
    Running setup.py clean for llvmlite
  Failed to build llvmlite
  Installing collected packages: llvmlite
    Running setup.py install for llvmlite: started
    Running setup.py install for llvmlite: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Running setup.py install for llvmlite did not run successfully.
    │ exit code: 1
    ╰─> [16 lines of output]
        running install
        /Users/oscardefelice/miniconda3/envs/tempr/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        running build
        got version from file /private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/llvmlite/_version.py {'version': '0.38.0', 'full': 'f0365b91ce1e1f74c68785c6d0067f32f89857d9'}
        running build_ext
        /Users/oscardefelice/miniconda3/envs/tempr/bin/python /private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py
        LLVM version... Traceback (most recent call last):
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 220, in <module>
            main()
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 214, in main
            main_posix('osx', '.dylib')
          File "/private/var/folders/gc/5dj19dqj64ddh8_sw790yqtr0000gn/T/pip-req-build-zd1dtsv2/ffi/build.py", line 134, in main_posix
            raise RuntimeError(msg) from None
        RuntimeError: Could not find a `llvm-config` binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help.
        error: command '/Users/oscardefelice/miniconda3/envs/tempr/bin/python' failed with exit code 1
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: legacy-install-failure
  
  × Encountered error while trying to install package.
  ╰─> llvmlite
  
  note: This is an issue with the package mentioned above, not pip.
  hint: See above for output from the failure.

I am looking around for this error, but I do not find anything useful.

While if I run poetry install I got the even less clear error

RuntimeError

   Unable to find installation candidates for tensorflow (2.9.1)

   at /usr/local/lib/python3.9/site-packages/poetry/installation/chooser.py:72 in choose_for
      68│ 
      69│             links.append(link)
      70│ 
      71│         if not links:
→  72│             raise RuntimeError(
     73│                 "Unable to find installation candidates for {}".format(package)
     74│             )
     75│ 
     76│         # Get the best link

[dimbleby]

duplicate #5752, #5920 tldr not a poetry problem

[mkniewallner]

I don't thing this has anything to do with Poetry, since the issue happens when building the wheel locally.

The issue itself is probably that you are using a Mac M1, which using arm64 architecture, but the version of llvmlite (a transitive dependency) requested by tensorflow doesn't provide wheels for this platform, meaning that you probably need to install some system dependencies.

I would suggest searching https://github.com/tensorflow/tensorflow issue tracker, and if nothing matches your issue, opening an issue there.

[oscar-defelice]

I don't thing this has anything to do with Poetry, since the issue happens when building the wheel locally.

The issue itself is probably that you are using a Mac M1, which using arm64 architecture, but the version of llvmlite (a transitive dependency) requested by tensorflow doesn't provide wheels for this platform, meaning that you probably need to install some system dependencies.

I opened an issue in llvmlite numba/llvmlite#868, but they say they have m1 wheels since a couple of months.

[dimbleby]

there is no built wheel at the version of llvmlite that you are trying to install though.

we've already been through this in the duplicate issues that I linked, please close

[oscar-defelice]

@dimbleby Thank you for your answer.

Can you please show me where I can find which version I should add to my project?
Because I have read the linked discussions carefully and I could not find any solution to my problem.
Furthermore, knowing the wheel issue, I have looked at llvmlite versions and apparently the one I am trying to install (llvmlite = "0.39.0") has it for M1. So I am confused.

[dimbleby]

the output you pasted shows that you are trying to install "0.38.0"

[oscar-defelice]

the output you pasted shows that you are trying to install "0.38.0"

Thank you. Same problem with 0.39.0

[dimbleby]

then there isn't a built wheel suitable for your environment at 0.39.0

[esc]

then there isn't a built wheel suitable for your environment at 0.39.0

This is not correct. There are pre-built wheels for llvmlite 0.39.0 for OSX ARM64 for Python 3.8, 3.9 and 3.10 available from PyPi:

https://pypi.org/project/llvmlite/#files

Is there anything that make these wheels unsuitable?

[dimbleby]

please show output in which poetry fails to install 0.39.0

[oscar-defelice]

The error is the same posted above if I run poetry add [email protected].
Now I managed to manually add llvmlite to my project by poetry add llvmlite --lock -vvv. This is the output, no error.

However, now when I run poetry install I got again the same error about not finding any candidate for tensorflow. no matter which version, I tried, 2.7.0, 2.7.1, 2.9.0, 2.9.1

[dimbleby]

The error is the same posted above if I run poetry add [email protected].
Now I managed to manually add llvmlite to my project by poetry add llvmlite --lock -vvv.

these statements are not consistent

the same error about not finding any candidate for tensorflow.

it's possible that you also have some #5970 hangover, maybe worth clearing your poetry cache and re-locking.

[oscar-defelice]

Ok, done.
Cleaned cache.

Now I have the following pyproject.toml

[tool.poetry]
name = "dataflows"
version = "0.1.0"
description = ""
authors = ["pthumerel <[email protected]>"]

[tool.poetry.dependencies]
python = ">=3.9, <3.10"
kedro = "0.18.1"
dask = {extras = ["distributed"], version = "^2021.12.0"}
pandas = "<1.4.0"
deap = "^1.3.1"
jupyter-client = "6.1.12"
pathos = "^0.2.8"
fsspec = "2021.11.1"
s3fs = "2021.11.1"
prophet = "^1.0.1"
tensorflow = "^2.9.1"
seaborn = "^0.11.2"


[tool.poetry.dev-dependencies]
jupyterlab = "^3.2.5"
tempr_ds = { path = "./tempr_ds/", develop = true }
kedro-viz = "^4.5.0"
dask-labextension = "^5.1.0"

# Jupyter Lab extension
jupyterlab-code-formatter = "^1.4.10"
isort = "^5.10.1"
black = "^21.12b0"

jupyterlab-system-monitor = "^0.8.0"
lckr-jupyterlab-variableinspector = "^3.0.9"
#jupyterlab-git = "^0.34.0"
python-crontab = "^2.6.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

and when I run poetry install

error: legacy-install-failure
  
  × Encountered error while trying to install package.
  ╰─> llvmlite
  
  note: This is an issue with the package mentioned above, not pip.
  hint: See above for output from the failure.
  

  at ~/miniconda3/envs/tempr/lib/python3.9/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│ 
      1197│         return decode(output)
      1198│ 
      1199│     def execute(self, bin, *args, **kwargs):

[oscar-defelice]

I have all of this into a dockerfile.

ARG BASE_IMAGE=python:3.9-buster
FROM $BASE_IMAGE

# install project requirements
# copy the files needed for installing requirements
RUN mkdir /dataflows

COPY .. /dataflows

# install poetry
RUN /usr/local/bin/python -m pip install --upgrade pip
RUN pip install poetry

WORKDIR /dataflows

# install python dependencies & remove the files
RUN poetry config virtualenvs.create false
RUN poetry install
RUN rm -rfd *

which (unsurprisingly) when I run docker build gives me

RuntimeError
#0 415.6 
#0 415.6   Unable to find installation candidates for tensorflow (2.9.1)
#0 415.6 
#0 415.6   at /usr/local/lib/python3.9/site-packages/poetry/installation/chooser.py:72 in choose_for
#0 415.6        68│ 
#0 415.6        69│             links.append(link)
#0 415.6        70│ 
#0 415.6        71│         if not links:
#0 415.6     →  72│             raise RuntimeError(
#0 415.6        73│                 "Unable to find installation candidates for {}".format(package)
#0 415.6        74│             )
#0 415.6        75│ 
#0 415.6        76│         # Get the best link
#0 415.6 
------

[dimbleby]

Your pyproject.toml is valid only on your machine, and it's indeed likely that the interesting bit is in tempr_ds since the rest doesn't cause llvmlite to be required at all.

Your dockerfile is valid only on your machine because it is copying things in that you have and no-one else does.

[oscar-defelice]

@dimbleby thank you very much.

However I deleted now tempr_ds and I am trying to run a minimal project just to add tensorflow, and I have the very same error.

1. I cannot add tensorflow dependency to the project with poetry add tensorflow.
2. If I add it by hand in the pyproject.toml then when I run poetry install I have the error stating
   Unable to find installation candidates for tensorflow (2.9.1)

Your dockerfile is valid only on your machine because it is copying things in that you have and no-one else does.

True, however it only copies a folder where there is the pyproject.toml and then sets such folder as working dir.

[dimbleby]

which of these files are you expecting to install? Is it possible that poetry is right and none of them are compatible with MacOS Monterey? There is no source distribution available.

(Not a mac user myself, but I see that you talk of 12.5, and the wheels say 10_14, sounds plausible to me that these are not compatible)

[dougmet]

Not sure if this helps but I've had some success using the tensorflow-macos library to get arm based wheels. It's a fork by Apple that supports M1. Inside poetry we used something like:

tensorflow = {version = ">=2.9.1", markers = "sys_platform != 'darwin'"}
tensorflow-macos = {version = ">=2.9.1", markers = "sys_platform == 'darwin'"}

which we're still tweaking so that it runs as expected on different platforms. This may require some system dependencies that I've forgotten about.

There's an Apple guide here although we didn't do step 1 and just installed tensorflow-macos and tensorflow-metal.

[dimbleby]

Thanks - per my last I suspect that the issue Unable to find installation candidates for tensorflow (2.9.1) was indeed simply that no such packages are available for the reporter's platform, and there's nothing much poetry can do about this.

If Apple are maintaining a version of tensorflow for this exact purpose then that does seem like a good bet.

Anyway OP seemed to lose interest in getting to the bottom of this, suggest closing.

[neersighted]

Closing as this is an issue with available wheels for the concrete versions Poetry is trying to install according to package constraints -- tensorflow-macos looks to be a suitable substitute for anyone running into this.

[SH2282000]

-- tensorflow-macos looks to be a suitable substitute for anyone running into this.
After a tensorflow-macos, everything works fine.

Thank you so much, that was it!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.