Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip install pyiceberg on windows require C++ to be installed #5901

Closed
djouallah opened this issue Oct 1, 2022 · 11 comments
Closed

pip install pyiceberg on windows require C++ to be installed #5901

djouallah opened this issue Oct 1, 2022 · 11 comments

Comments

@djouallah
Copy link

Apache Iceberg version

main (development)

Query engine

No response

Please describe the bug 🐞

when I tired to install pip install pyiceberg on windows 10, it did complaining about missing C++ files, I was expect a binary distribution not compiling the source myself ? is this by design
@samredai
Copy link
Collaborator

samredai commented Oct 2, 2022

I believe this is because of the pyarrow dependency. In the pyarrow docs they include tips for getting around installation issues on windows.

@djouallah
Copy link
Author

I have pyarrow installed already?

@Fokko
Copy link
Contributor

Fokko commented Oct 2, 2022

@djouallah Thanks for reaching out and creating the issue.

Could you share the command that you use to install PyIceberg? For example, pip3 install pyiceberg[pyarrow] would install it with PyArrow support, or pip3 install pyiceberg[s3fs] would install it with the s3fs backend. Knowing this would help us to reproduce it.

Could you also share the error that you're seeing? Just the output of the console would be helpful 👍🏻

@djouallah
Copy link
Author

PS C:\Users\mimoune.djouallah\Desktop\TPC-H-SF10> pip3 install pyiceberg[pyarrow]

Collecting pyiceberg[pyarrow]
Using cached pyiceberg-0.1.0-py3-none-any.whl (276 kB)
Requirement already satisfied: rich<13.0.0,>=12.5.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (12.5.1)
Requirement already satisfied: click<9.0.0,>=8.1.3 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (8.1.3)
Collecting zstandard<0.19.0,>=0.18.0
Using cached zstandard-0.18.0-cp310-cp310-win_amd64.whl (611 kB)
Collecting fsspec==2022.8.2
Using cached fsspec-2022.8.2-py3-none-any.whl (140 kB)
Collecting mmh3<4.0.0,>=3.0.0
Using cached mmh3-3.0.0.tar.gz (10 kB)
Preparing metadata (setup.py) ... done
Collecting pyyaml<7.0.0,>=6.0.0
Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Requirement already satisfied: pydantic<2.0.0,>=1.10.2 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (1.10.2)Requirement already satisfied: requests<3.0.0,>=2.28.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (2.28.1)Requirement already satisfied: pyarrow<10.0.0,>=9.0.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (9.0.0)
Requirement already satisfied: colorama in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from click<9.0.0,>=8.1.3->pyiceberg[pyarrow]) (0.4.5)
Requirement already satisfied: numpy>=1.16.6 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyarrow<10.0.0,>=9.0.0->pyiceberg[pyarrow]) (1.23.1)
Requirement already satisfied: typing-extensions>=4.1.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pydantic<2.0.0,>=1.10.2->pyiceberg[pyarrow]) (4.3.0)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (1.26.11)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (2022.6.15)
Requirement already satisfied: idna<4,>=2.5 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (3.3)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from rich<13.0.0,>=12.5.1->pyiceberg[pyarrow]) (0.9.1)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from rich<13.0.0,>=12.5.1->pyiceberg[pyarrow]) (2.13.0)
Using legacy 'setup.py install' for mmh3, since package 'wheel' is not installed.
Installing collected packages: mmh3, zstandard, pyyaml, fsspec, pyiceberg
Running setup.py install for mmh3 ... error
error: subprocess-exited-with-error

× Running setup.py install for mmh3 did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
running install
C:\Users\mimoune.djouallah\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
building 'mmh3' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mmh3

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

@Fokko
Copy link
Contributor

Fokko commented Oct 3, 2022

Thanks @djouallah for sharing. Could you try to install wheel (pip3 install wheel), and then install PyIceberg?

I noticed this from the log:

Using legacy 'setup.py install' for mmh3, since package 'wheel' is not installed.

This means that it will fall back on installing it from the source, which will mean that it will try to compile the native extension.

@djouallah
Copy link
Author

PS C:\Users\mimoune.djouallah\Desktop\TPC-H-SF10> pip3 install wheel
Requirement already satisfied: wheel in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (0.37.1)
PS C:\Users\mimoune.djouallah\Desktop\TPC-H-SF10> pip3 install pyiceberg[pyarrow]
Collecting pyiceberg[pyarrow]
Using cached pyiceberg-0.1.0-py3-none-any.whl (276 kB)
Collecting mmh3<4.0.0,>=3.0.0
Using cached mmh3-3.0.0.tar.gz (10 kB)
Preparing metadata (setup.py) ... done
Collecting pyyaml<7.0.0,>=6.0.0
Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Requirement already satisfied: click<9.0.0,>=8.1.3 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (8.1.3)
Collecting zstandard<0.19.0,>=0.18.0
Using cached zstandard-0.18.0-cp310-cp310-win_amd64.whl (611 kB)
Requirement already satisfied: pydantic<2.0.0,>=1.10.2 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (1.10.2)
Requirement already satisfied: rich<13.0.0,>=12.5.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (12.5.1)
Requirement already satisfied: requests<3.0.0,>=2.28.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (2.28.1)
Collecting fsspec==2022.8.2
Using cached fsspec-2022.8.2-py3-none-any.whl (140 kB)
Requirement already satisfied: pyarrow<10.0.0,>=9.0.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyiceberg[pyarrow]) (9.0.0)
Requirement already satisfied: colorama in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from click<9.0.0,>=8.1.3->pyiceberg[pyarrow]) (0.4.5)Requirement already satisfied: numpy>=1.16.6 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pyarrow<10.0.0,>=9.0.0->pyiceberg[pyarrow]) (1.23.1)
Requirement already satisfied: typing-extensions>=4.1.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from pydantic<2.0.0,>=1.10.2->pyiceberg[pyarrow]) (4.3.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (1.26.11)
Requirement already satisfied: idna<4,>=2.5 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (2022.6.15)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from requests<3.0.0,>=2.28.1->pyiceberg[pyarrow]) (2.1.1)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from rich<13.0.0,>=12.5.1->pyiceberg[pyarrow]) (2.13.0)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in c:\users\mimoune.djouallah\appdata\local\programs\python\python310\lib\site-packages (from rich<13.0.0,>=12.5.1->pyiceberg[pyarrow]) (0.9.1)
Building wheels for collected packages: mmh3
Building wheel for mmh3 (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [5 lines of output]
running bdist_wheel
running build
running build_ext
building 'mmh3' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for mmh3
Running setup.py clean for mmh3
Failed to build mmh3
Installing collected packages: mmh3, zstandard, pyyaml, fsspec, pyiceberg
Running setup.py install for mmh3 ... error
error: subprocess-exited-with-error

× Running setup.py install for mmh3 did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
running install
C:\Users\mimoune.djouallah\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
building 'mmh3' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mmh3

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure

@djouallah
Copy link
Author

I think the issue is with mmh3

PS C:\Users\mimoune.djouallah\Desktop\TPC-H-SF10> pip3 install mmh3
Collecting mmh3
Using cached mmh3-3.0.0.tar.gz (10 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: mmh3
Building wheel for mmh3 (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [5 lines of output]
running bdist_wheel
running build
running build_ext
building 'mmh3' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for mmh3
Running setup.py clean for mmh3
Failed to build mmh3
Installing collected packages: mmh3
Running setup.py install for mmh3 ... error
error: subprocess-exited-with-error

× Running setup.py install for mmh3 did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
running install
C:\Users\mimoune.djouallah\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
building 'mmh3' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mmh3

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
PS C:\Users\mimoune.djouallah\Desktop\TPC-H-SF10>

@Fokko
Copy link
Contributor

Fokko commented Oct 4, 2022

@djouallah The pre-compiled wheel is not available for Python 3.10. It is available for Python <=3.9. There is a PR available for this: hajimes/mmh3#35 but has yet to be merged.

@djouallah
Copy link
Author

it is seems mmh3 is not supported, is there an alternative solution please ?

Fokko added a commit to Fokko/iceberg that referenced this issue Oct 29, 2022
@Fokko
Python: Replace mmh3 with mmhash3
45cbdb1 
The mmh3 project is abandoned, and not being maintained.

This results in that there are no wheels (compiled binaries) available
for newer versions of Python (3.10 and 3.11). I've forked the repo,
and created a new release with the latest binaries.

Solves apache#5901
Fokko added a commit to Fokko/iceberg that referenced this issue Oct 29, 2022
@Fokko
Python: Replace mmh3 with mmhash3
b4ac28d 
The mmh3 project is abandoned, and not being maintained.

This results in that there are no wheels (compiled binaries) available
for newer versions of Python (3.10 and 3.11). I've forked the repo,
and created a new release with the latest binaries.

Solves apache#5901
@Fokko Fokko mentioned this issue Oct 29, 2022
Merged
Fokko added a commit to Fokko/iceberg that referenced this issue Oct 29, 2022
@Fokko
Python: Replace mmh3 with mmhash3
50e6014 
The mmh3 project is abandoned, and not being maintained.

This results in that there are no wheels (compiled binaries) available
for newer versions of Python (3.10 and 3.11). I've forked the repo,
and created a new release with the latest binaries.

Solves apache#5901
Fokko added a commit to Fokko/iceberg that referenced this issue Nov 3, 2022
@Fokko
Python: Replace mmh3 with mmhash3
f2fbcf3 
The mmh3 project is abandoned, and not being maintained.

This results in that there are no wheels (compiled binaries) available
for newer versions of Python (3.10 and 3.11). I've forked the repo,
and created a new release with the latest binaries.

Solves apache#5901
rdblue pushed a commit that referenced this issue Nov 6, 2022
@Fokko
Python: Replace mmh3 with mmhash3 (#6076)
0342758 
The mmh3 project is abandoned, and not being maintained.

Solves #5901
@djouallah
Copy link
Author

thanks, it works now with 0.2 release

@Fokko
Copy link
Contributor

Fokko commented Dec 9, 2022

Awesome, thanks for letting us know! @djouallah

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Fokko @djouallah @samredai