Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UnicodeDecodeError in Masonry with MSYS2 #1027

Closed
3 tasks done
danyeaw opened this issue Apr 10, 2019 · 11 comments
Closed
3 tasks done

UnicodeDecodeError in Masonry with MSYS2 #1027

danyeaw opened this issue Apr 10, 2019 · 11 comments

Comments

@danyeaw
Copy link
Contributor

danyeaw commented Apr 10, 2019

Issue

When trying to pip install my app I am getting a UnicodeDecodeError.

$ pip3 install --user .
Processing c:/tools/msys64/home/dyeaw/gaphor
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... error
    Complete output from command C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr:
    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 64, in prepare_metadata_for_build_wheel
        hook = backend.prepare_metadata_for_build_wheel
    AttributeError: module 'poetry.masonry.api' has no attribute 'prepare_metadata_for_build_wheel'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 207, in <module>
        main()
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 67, in prepare_metadata_for_build_wheel
        config_settings)
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 95, in _get_wheel_metadata_from_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\api.py", line 40, in build_wheel
        poetry, SystemEnv(Path(sys.prefix)), NullIO(), Path(wheel_directory)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 48, in make_in
        wb = WheelBuilder(poetry, env, io, target_dir=directory, original=original)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 38, in __init__
        super(WheelBuilder, self).__init__(poetry, env, io)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\builder.py", line 42, in __init__
        self._meta = Metadata.from_package(self._package)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\metadata.py", line 50, in from_package
        meta.description = f.read()
      File "C:/tools/msys64/mingw64/lib/python3.7\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 7221: character maps to <undefined>

    ----------------------------------------
Command "C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr" failed with error code 1 in C:/Users/dyeaw/AppData/Local/Temp/pip-req-build-clnw4_9m

It looks like Poetry is not using an encoding when it calls open() to load the description.

I am opening the issue here after exploring the cause in MSYS2 first: msys2/MINGW-packages#5155
@drunkwcodes
Copy link

Save your pyproject.toml in utf-8 encoding again should be alright?

To handle different encodings, it needs to depend on chardet without a guarentee resolving this problem.

Reference: https://stackoverflow.com/questions/436220/how-to-determine-the-encoding-of-text

@danyeaw
Copy link
Contributor Author

danyeaw commented Apr 10, 2019

@drunkwcodes My pyproject.toml, and the rest of my project is in UTF-8. The issue is that Poetry is calling open() with no encoding specified, and the default encoding for Windows is cp1252.

@danyeaw danyeaw mentioned this issue Apr 11, 2019
Closed
2 tasks
@drunkwcodes
Copy link

drunkwcodes commented Apr 11, 2019 •
edited
Loading

Sorry. Here it is. Please try to change msys2's encoding setting to utf-8.
Or use chcp 65001 command.

https://stackoverflow.com/questions/28481410/how-to-set-console-encoding-in-msys

@danyeaw
Copy link
Contributor Author

danyeaw commented Apr 11, 2019

@drunkwcodes Thanks for trying to research possible solutions. The correct default/preferred encoding of the contents of files for Windows users is cp1252. This is what is returned when I call locale.getpreferredencoding(). It is not a solution for me to change my default encoding. This issue here is that Poetry is opening pyproject.toml, which should be UTF-8 for all users, without specifying the encoding to use.

@drunkwcodes
Copy link

@danyeaw Sorry for not having a solution for you to change your default encoding, sir. What a lovely night.

@joar
Copy link

joar commented Apr 12, 2019

I believe #1030 is related to this issue. I don't think it's safe to assume that the locale will always be set to UTF-8 on systems where poetry (or packages using poetry, such as pendulum) might be installed.

@joar joar mentioned this issue Apr 12, 2019
Closed
3 tasks
@vlcinsky
Copy link
Contributor

vlcinsky commented May 7, 2019

We (me and @rhorenov) had problem to install pendulum==2.0.4 on MS Windows with encoding windows-1250.

If the package was to be built (e.g. pendulum), it failed on special characters in metadata generated by poetry as it was opening files for build using default console encoding.

Installing packages from wheels worked well.

I am going to create PR for master to fix this issue (adding explicit encoding where files are open for writing in text mode).

vlcinsky added a commit to vlcinsky/poetry that referenced this issue May 7, 2019
@vlcinsky
Fix encoding for files created for building package (issue python-poe…
0490027 
…try#1027)

This is minimal set of modifications, which fixed the problem.

There are still more places, where files are open in text mode
without explicit encoding, thus possibly failing on systems not using
UTF-8 console encoding.
@vlcinsky vlcinsky mentioned this issue May 7, 2019
Closed
3 tasks
@cjerdonek cjerdonek mentioned this issue Jun 19, 2019
Closed
sdispater pushed a commit that referenced this issue Jun 30, 2019
@vlcinsky @sdispater
Review of (explicit) encoding for files being created in text mode (#…
e9a13f8 
…1088)

* Fix encoding for files created for building package (issue #1027)

This is minimal set of modifications, which fixed the problem.

There are still more places, where files are open in text mode
without explicit encoding, thus possibly failing on systems not using
UTF-8 console encoding.

* test_api.py use explicit encoding for files.

* get-poetry.py: fix explicit (utf-8) encoding for files open in text mode

* tests: fix explicit (utf-8) encoding for files open in text mode

* poetry init: use explicit utf-8 for created pyproject.toml

* installed: use explicit utf-8 for text files

* layouts: use explicit utf-8 for open text files

* spdx: explicit utf-8 for open text files

* pypi_repository: explicit utf-8 for open text files

* explicit utf-8 for json

* explicit utf-8 for metadata

* explicit utf-8 for requires.txt

* open files with explicit encoding in py27

* fix creation of poetry.bat on Windows

* Make get-poetry.py compatible to py2/3

* Blacked two files to pass linter test

* remove extranous encoding related comments

* rewert broken move of import related to ansible

* removed extraneous comments about opening files in UTF-8

* rewert ansible.release import (keep order of vars unchanged)

* Revert to original content (as it comes from external project)

* all vendored setup.py files in fixtures reverted back to org
@vlcinsky
Copy link
Contributor

Can @danyeaw or anyone else check, that the latest poetry version 0.12.17 is not not suffering this problem?

@danyeaw
Copy link
Contributor Author

danyeaw commented Jul 10, 2019

Closed by commit e9a13f8 👍

@vlcinsky
Copy link
Contributor

Greate @danyeaw

Can you close the issue? I do not seem to have such option.

@danyeaw danyeaw closed this as completed Jul 11, 2019
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 3, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix UnicodeDecodeError caused by opening README with no encoding danyeaw/poetry
4 participants
@joar @vlcinsky @danyeaw @drunkwcodes