pip 19.1.1 fails to install pendulum 2.0.4

[smtakeda]

Environment

• pip version: 19.1.1
• Python version: 3.6.8
• OS: Windows 10 Enterprise

Package Version

---

pip 19.1.1
setuptools 41.0.1

Description
pip 19.1.1 / 19.0 fails to install pendulum 2.0.4.
pip 18.1 works well.

No problem on Linux.

Expected behavior
pendulum package is installed:

>pip install pendulum
Collecting pendulum
  Using cached https://files.pythonhosted.org/packages/5b/57/71fc910edcd937b72aa0ef51c8f5734fbd8c011fa1480fce881433847ec8/pendulum-2.0.4.tar.gz
  Installing build dependencies ... done
Collecting python-dateutil<3.0,>=2.6 (from pendulum)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting pytzdata>=2018.3 (from pendulum)
  Using cached https://files.pythonhosted.org/packages/a7/90/c9a148b4c98ccf88765ab7399debd38a33660109c9ba23706f560109b6e1/pytzdata-2019.1-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0,>=2.6->pendulum)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytzdata, pendulum
  Running setup.py install for pendulum ... done
Successfully installed pendulum-2.0.4 python-dateutil-2.8.0 pytzdata-2019.1 six-1.12.0

How to Reproduce
Open a cmd and run the following.

python -m venv test
call test\scripts\activate
python -m pip install -U pip
pip install pendulum

Output

>python -m venv test
>call test\scripts\activate
>python -m pip install -U pip  # updgrade to 19.1.1
Collecting pip
  Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.1.1
>pip install pendulum
Collecting pendulum
  Using cached https://files.pythonhosted.org/packages/5b/57/71fc910edcd937b72aa0ef51c8f5734fbd8c011fa1480fce881433847ec8/pendulum-2.0.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
ERROR: Exception:
Traceback (most recent call last):
  File "c:\users\shige\test\lib\site-packages\pip\_internal\cli\base_command.py", line 178, in main
    status = self.run(options, args)
  File "c:\users\shige\test\lib\site-packages\pip\_internal\commands\install.py", line 352, in run
    resolver.resolve(requirement_set)
  File "c:\users\shige\test\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\users\shige\test\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\shige\test\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "c:\users\shige\test\lib\site-packages\pip\_internal\operations\prepare.py", line 362, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "c:\users\shige\test\lib\site-packages\pip\_internal\operations\prepare.py", line 171, in prep_for_dist
    self.req.prepare_metadata()
  File "c:\users\shige\test\lib\site-packages\pip\_internal\req\req_install.py", line 553, in prepare_metadata
    metadata_name = canonicalize_name(self.metadata["Name"])
  File "c:\users\shige\test\lib\site-packages\pip\_internal\req\req_install.py", line 674, in metadata
    self._metadata = get_metadata(self.get_dist())
  File "c:\users\shige\test\lib\site-packages\pip\_internal\utils\packaging.py", line 48, in get_metadata
    metadata = dist.get_metadata('METADATA')
  File "c:\users\shige\test\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1420, in get_metadata
    return value.decode('utf-8') if six.PY3 else value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 185: invalid continuation byte

[smtakeda]

hmm, looks like author's name includes an invalid utf-8 character.
S\xe9bastien

[smtakeda]

pendulum's PKG-INFO include a valid UTF-8 character, c3 c9. I suspect pip mistakenly converts it to iso-8859-1.

>od -t x1 -c PKG-INFO
0000000    4d  65  74  61  64  61  74  61  2d  56  65  72  73  69  6f  6e
           M   e   t   a   d   a   t   a   -   V   e   r   s   i   o   n
...
0000200    75  74  68  6f  72  3a  20  53  c3  a9  62  61  73  74  69  65
           u   t   h   o   r   :       S   é  **   b   a   s   t   i   e

[prokopst]

A colleague of mine has just hit the same issue.

@sdispater is owner of pendulum, I guess he should be aware of the problem.

[sladkovm]

The same error here. Windows 10.

[g-berthiaume]

The same error here. Windows 10.

$ pip --version
pip 19.1.1 from mydir (python 3.7)

[HWiese1980]

Confirmed, same here, Windows 10

[andrewtmendoza]

As a workaround, you can try installing via conda-forge if you are using Anaconda:

https://anaconda.org/conda-forge/pendulum

[HWiese1980]

Yes, that works. But it's a bit uncomfortable if you don't normally use Anaconda. I mean, Anaconda is great. No question about it. Especially on Windows. So it's a valid workaround. Still it's a bug that should be fixed.

[cjerdonek]

@smtakeda Can you try the install command with verbose logging?

$ pip -vv install pendulum

Also, this is from pendulum's pyproject.toml:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

So I wonder if this is an issue with poetry. Like, here are some still-open issues on poetry's tracker related to encodings:

• poetry 0.12.12 breaks pendulum install on Python 2.7 python-poetry/poetry#1030
• UnicodeDecodeError in Masonry with MSYS2 python-poetry/poetry#1027
• Review of (explicit) encoding for files being created in text mode python-poetry/poetry#1087

I wonder what pip can do to spot this issue earlier (or help resolve it), and help figure out what code is responsible for generating the bad file (e.g. if pip isn't at fault).

[cjerdonek]

I just filed this setuptools (pkg_resources) issue, FYI, which should help with giving a better error message: pypa/setuptools#1790

That won't go all the way though, IMO, in making instances of this issue easier to diagnose.

[cjerdonek]

FYI, yesterday I posted the following PR to setuptools which would improve the UnicodeDecodeError seen in pip: pypa/setuptools#1791
It would cause the path to the errant metadata file to be included in the error message text. That way it will be easier in the future to track down what package / tool is causing the error.

[cjerdonek]

FYI, my setuptools PR that adds to the exception text the path to the problematic file has now been merged: pypa/setuptools#1791

[chrahunt]

I'm not sure we're waiting for any response, just investigation as to how the valid utf-8 character in PKG-INFO got turned into an invalid character in the data that pkg_resources attempted to decode.

[cjerdonek]

@chrahunt I put "awaiting response" since I was waiting for a reply to this question in my comment:

Can you try the install command with verbose logging?

It would provide more info to help diagnose the root cause.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[sfc-gh-stakeda]

Sorry for late reply.. Anyway I attempted to reproduce this but no longer happening:

C:\Users\shige>call test_pendulum\scripts\activate

(test_pendulum) C:\Users\shige>python -m pip install -U pip==19.1.1
Collecting pip==19.1.1
  Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.1.1

(test_pendulum) C:\Users\shige>pip install -U setuptools==41.0.1
Collecting setuptools==41.0.1
  Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.6.2
    Uninstalling setuptools-40.6.2:
      Successfully uninstalled setuptools-40.6.2
Successfully installed setuptools-41.0.1
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

(test_pendulum) C:\Users\shige>pip install pendulum==2.0.4
Collecting pendulum==2.0.4
Collecting pytzdata>=2018.3 (from pendulum==2.0.4)
  Using cached https://files.pythonhosted.org/packages/7f/f9/cdd39831b0465285c02ed90cfbf0db25bb951cb2a35ded0e69222375bea3/pytzdata-2019.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0,>=2.6 (from pendulum==2.0.4)
  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0,>=2.6->pendulum==2.0.4)
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Installing collected packages: pytzdata, six, python-dateutil, pendulum
Successfully installed pendulum-2.0.4 python-dateutil-2.8.1 pytzdata-2019.3 six-1.13.0
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

(test_pendulum) C:\Users\shige>pip list
Package         Version
--------------- -------
pendulum        2.0.4
pip             19.1.1
python-dateutil 2.8.1
pytzdata        2019.3
setuptools      41.0.1
six             1.13.0
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

I'm sure this was reproducible when the issue was reported, because both my local Windows vm and Appveyor had the same issue.

Thank you for looking into this anyway.

[mriffey]

Same here. Happened on 2 different machines previously. Problem is gone now. Thank you to whoever fixed it. Mark