pip install cwltool failed with mypy-extension==0.4.0 #872

inutano · 2018-08-17T10:14:17Z

Expected Behavior

pip install cwltool will give you cwltool

Actual Behavior

pip install cwltool failed with error of Downloading/unpacking mypy-extensions.

I guess the recent version of mypy-extension==0.4.0 will cause the problem, since pip install mypy-extension==0.3.0 followed by pip install cwltool goes fine.

Full Traceback

This Dockerfile

FROM debian:jessie-slim
RUN apt-get update -qq
RUN apt-get install -y python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev
RUN pip install cwltool

Failed.

Step 4/4 : RUN pip install cwltool
 ---> Running in c7d559c034e2
Downloading/unpacking cwltool
Downloading/unpacking bagit>=1.6.4 (from cwltool)
  Downloading bagit-1.7.0-py2.py3-none-any.whl
Downloading/unpacking subprocess32>=3.5.0 (from cwltool)
  Running setup.py (path:/tmp/pip-build-67BGbi/subprocess32/setup.py) egg_info for package subprocess32
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)

    no previously-included directories found matching 'build'
    no previously-included directories found matching 'dist'
    no previously-included directories found matching 'config.*'
    no previously-included directories found matching '*.pyc'
Downloading/unpacking mypy-extensions (from cwltool)
  Downloading mypy_extensions-0.4.0.tar.gz
  Running setup.py (path:/tmp/pip-build-67BGbi/mypy-extensions/setup.py) egg_info for package mypy-extensions
    error in mypy_extensions setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
    Complete output from command python setup.py egg_info:
    error in mypy_extensions setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-67BGbi/mypy-extensions
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c pip install cwltool' returned a non-zero code: 1

Your Environment

cwltool version: without version specification in pip install

The text was updated successfully, but these errors were encountered:

manabuishii · 2018-08-17T10:18:14Z

mypy_extensions 0.4.0 is released yesterday (I think 2018-08-16 ??).
0.3.0 released 2017-07-07

mypy_extensions · PyPI

sersorrel · 2018-08-17T10:52:29Z

The relevant change in mypy-extensions was python/mypy@d9581fe, which added a dependency on the typing module for Python 3.4 and below:

install_requires=[
    'typing >= 3.5.3; python_version < "3.5"',
]

Unfortunately, setuptools doesn't support environment markers (like python_version) in install_requires until version 20.5, which was released in 2016 – the version of setuptools packaged in Debian Jessie is only version 5.5.1.

However, setuptools does support environment markers in extras_require since version 0.7, so it should be possible to update mypy-extensions to work around this.

mr-c · 2018-08-17T11:11:32Z

@inutano Can you confirm that using recent setuptools & pip fixes the issue?

inutano · 2018-08-17T15:14:48Z

Thank you @anowlcalledjosh! I've confirmed that the installation succeeded by updating setuptools beforehand:

root@b1a8f220f1cf:/data# pip show pip
---
Name: pip
Version: 1.5.6
Location: /usr/lib/python2.7/dist-packages
Requires:
root@ca98f3b85735:/data# pip show setuptools
---
Name: setuptools
Version: 5.5.1
Location: /usr/lib/python2.7/dist-packages
Requires:
root@b1a8f220f1cf:/data# pip install setuptools --upgrade
Downloading/unpacking setuptools from https://files.pythonhosted.org/packages/ff/f4/385715ccc461885f3cedf57a41ae3c12b5fec3f35cce4c8706b1a112a133/setuptools-40.0.0-py2.py3-none-any.whl#sha256=d68abee4eed409fbe8c302ac4d8429a1ffef912cd047a903b5701c024048dd49
  Downloading setuptools-40.0.0-py2.py3-none-any.whl (567kB): 567kB downloaded
Installing collected packages: setuptools
  Found existing installation: setuptools 5.5.1
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed setuptools
Cleaning up...
root@b1a8f220f1cf:/data# pip show setuptools
---
Name: setuptools
Version: 40.0.0
Location: /usr/local/lib/python2.7/dist-packages
Requires:
root@ca98f3b85735:/data# pip install cwltool
...
root@ca98f3b85735:/data# which cwltool
/usr/local/bin/cwltool

However, cwltool --help failed after the installation:

root@ca98f3b85735:/data# cwltool --help
Traceback (most recent call last):
  File "/usr/local/bin/cwltool", line 7, in <module>
    from cwltool.main import run
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 28, in <module>
    from . import command_line_tool, workflow
  File "/usr/local/lib/python2.7/dist-packages/cwltool/command_line_tool.py", line 28, in <module>
    from .docker import DockerCommandLineJob
  File "/usr/local/lib/python2.7/dist-packages/cwltool/docker.py", line 19, in <module>
    from .job import ContainerCommandLineJob
  File "/usr/local/lib/python2.7/dist-packages/cwltool/job.py", line 24, in <module>
    from prov.model import PROV
  File "/usr/local/lib/python2.7/dist-packages/prov/model.py", line 24, in <module>
    from prov.constants import *
  File "/usr/local/lib/python2.7/dist-packages/prov/constants.py", line 5, in <module>
    from prov.identifier import Namespace
  File "/usr/local/lib/python2.7/dist-packages/prov/identifier.py", line 10, in <module>
    @six.python_2_unicode_compatible
AttributeError: 'module' object has no attribute 'python_2_unicode_compatible'

I've found pip install six --upgrade fixes this:

root@ca98f3b85735:/data# pip show six
---
Name: six
Version: 1.8.0
Location: /usr/lib/python2.7/dist-packages
Requires:
root@ca98f3b85735:/data# pip install six --upgrade
Downloading/unpacking six from https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl#sha256=832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb
  Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six
  Found existing installation: six 1.8.0
    Not uninstalling six at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed six
Cleaning up...
root@ca98f3b85735:/data# pip show six
---
Name: six
Version: 1.11.0
Location: /usr/local/lib/python2.7/dist-packages
Requires:
root@ca98f3b85735:/data# cwltool --help
usage: cwltool [-h] [--basedir BASEDIR] [--outdir OUTDIR] [--parallel]
               [--preserve-environment ENVVAR | --preserve-entire-environment]
               [--rm-container | --leave-container] [--record-container-id]
...

Seems all went fine, but another error happens when I tried to validate a CWL workflow:

oot@ca98f3b85735:/data# cwltool --validate fastqc_wf.cwl
Traceback (most recent call last):
  File "/usr/local/bin/cwltool", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 713, in run
    sys.exit(main(*args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 427, in main
    _logger.info(versionfunc())
  File "/usr/local/lib/python2.7/dist-packages/cwltool/utils.py", line 45, in versionstring
    pkg = pkg_resources.require("cwltool")
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ruamel.ordereddict' distribution was not found and is required by ruamel.yaml

Google saved me again:

root@b1a8f220f1cf:/data# pip show ruamel.ordereddict
root@b1a8f220f1cf:/data# pip install ruamel.ordereddict
Downloading/unpacking ruamel.ordereddict
  Downloading ruamel.ordereddict-0.4.13.tar.gz (57kB): 57kB downloaded
  Running setup.py (path:/tmp/pip-build-Cd5ayD/ruamel.ordereddict/setup.py) egg_info for package ruamel.ordereddict
Installing collected packages: ruamel.ordereddict
  Running setup.py install for ruamel.ordereddict
    building '_ordereddict' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c ordereddict.c -o build/temp.linux-x86_64-2.7/ordereddict.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/ordereddict.o -o build/lib.linux-x86_64-2.7/_ordereddict.so
    Installing /usr/local/lib/python2.7/dist-packages/ruamel.ordereddict-0.4.13-py2.7-nspkg.pth
  Could not find .egg-info directory in install record for ruamel.ordereddict
Successfully installed ruamel.ordereddict
Cleaning up...
root@b1a8f220f1cf:/data# cwltool --validate fastqc_wf.cwl
/usr/local/bin/cwltool 1.0.20180809224403
Resolved 'fastqc_wf.cwl' to 'file:///data/fastqc_wf.cwl'
Tool definition failed initialization:
('https://raw.githubusercontent.com/pitagora-galaxy/cwl/master/tools/download-sra/download-sra.cwl', AttributeError("'HTTPResponse' object has no attribute 'chunked'",))

😢

root@b1a8f220f1cf:/data# pip show requests
---
Name: requests
Version: 2.4.3
Location: /usr/lib/python2.7/dist-packages
Requires:
root@b1a8f220f1cf:/data# pip install requests --upgrade
Downloading/unpacking requests from https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl#sha256=63b52e3c866428a224f97cab011de738c36aec0185aa91cfacd418b5d58911d1
  Downloading requests-2.19.1-py2.py3-none-any.whl (91kB): 91kB downloaded
Downloading/unpacking idna<2.8,>=2.5 (from requests)
  Downloading idna-2.7-py2.py3-none-any.whl (58kB): 58kB downloaded
Downloading/unpacking certifi>=2017.4.17 (from requests)
  Downloading certifi-2018.8.13-py2.py3-none-any.whl (146kB): 146kB downloaded
Downloading/unpacking chardet<3.1.0,>=3.0.2 (from requests)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB): 133kB downloaded
Downloading/unpacking urllib3<1.24,>=1.21.1 (from requests)
  Downloading urllib3-1.23-py2.py3-none-any.whl (133kB): 133kB downloaded
Installing collected packages: requests, idna, certifi, chardet, urllib3
  Found existing installation: requests 2.4.3
    Not uninstalling requests at /usr/lib/python2.7/dist-packages, owned by OS
  Found existing installation: chardet 2.3.0
    Not uninstalling chardet at /usr/lib/python2.7/dist-packages, owned by OS
  Found existing installation: urllib3 1.9.1
    Not uninstalling urllib3 at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed requests idna certifi chardet urllib3
Cleaning up...
root@b1a8f220f1cf:/data# cwltool --validate fastqc_wf.cwl
/usr/local/bin/cwltool 1.0.20180809224403
Resolved 'fastqc_wf.cwl' to 'file:///data/fastqc_wf.cwl'
Workflow checker warning:
fastqc_wf.cwl:7:3:  Source 'repo' of type ["null", "string"] may be incompatible
fastqc_wf.cwl:24:7:   with sink 'repo' of type "string"
Tool definition is valid

looks like it worked. wooo.

So far what I'm wondering is if there's a smarter way to specify minimum requirement versions of these python package dependencies. I'm not an expert for python package, but how do we improve this pip experience?

sersorrel · 2018-08-17T15:35:06Z

wrt six: cwltool depends on six >= 1.8.0, but python_2_unicode_compatible was added in 1.9.0, so cwltool should be updated to depend on a newer version of six.

Apparently the HTTPResponse.chunked issue is because of Requests' urllib3 dependency, but I couldn't find out which version it was actually added in.

ruamel.ordereddict: no idea.

Do things go more smoothly if you pip install -U pip setuptools wheel before installing cwltool?

inutano · 2018-08-18T05:13:50Z

FROM debian:jessie-slim
RUN apt-get update -qq
RUN apt-get install -y python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev
RUN pip install -U pip setuptools wheel
RUN pip install cwltool

Building this Dockerfile successfully finished and installed cwltool, but has the same error of six and requests when I try cwltool --help or cwltool --validate. Hmm.

I found an error message while pip install cwltool, I have no idea if this is related to the issue though:

prov 1.5.1 has requirement six>=1.9.0, but you'll have six 1.8.0 which is incompatible.

inutano · 2018-08-18T05:16:51Z

I've also confirmed that FROM ubuntu:14.04 has the same issue, while cwltool installed by Dockerfile below has no problem:

FROM ubuntu:16.04
RUN apt-get update -y && apt-get install -y python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev
RUN pip install cwltool

sersorrel · 2018-08-20T08:50:20Z

The problem is basically that Pip doesn't actually have a proper dependency resolver yet: pypa/pip#988, so Pip sees that cwltool depends on six >=1.8.0 and decides that the installed version of six (1.8.0) is ok, but when it looks at prov's dependencies, it's not clever enough to realise that it should install six 1.9.0 instead.

(This is getting a bit sidetracked from the original issue of "mypy-extensions 0.4.0 doesn't work with old setuptools", which I've just submitted a PR to fix.)

mr-c · 2018-08-20T12:37:45Z

A PR to bump up cwltool's six dependency (with a comment) would be appreciated

cwltool itself doesn't depend on six 1.9.0, but prov does, and having matching dependencies helps Pip find a correct set of packages to install (see common-workflow-language#872).

cwltool itself doesn't depend on requests 2.6.1, but CacheControl does, and having matching dependencies helps Pip find a correct set of packages to install (see common-workflow-language#872).

mr-c · 2018-08-21T10:51:09Z

Fixed in #877

cwltool itself doesn't depend on six 1.9.0, but prov does, and having matching dependencies helps Pip find a correct set of packages to install (see #872).

cwltool itself doesn't depend on requests 2.6.1, but CacheControl does, and having matching dependencies helps Pip find a correct set of packages to install (see #872).

sersorrel mentioned this issue Aug 20, 2018

Use extras_require over install_requires in mypy_extensions python/mypy#5511

Closed

sersorrel mentioned this issue Aug 20, 2018

Bump six and requests dependency versions #877

Merged

mr-c closed this as completed Aug 21, 2018

stain pushed a commit that referenced this issue Aug 21, 2018

Bump dependency on six from 1.8.0 to 1.9.0

e683fd8

cwltool itself doesn't depend on six 1.9.0, but prov does, and having matching dependencies helps Pip find a correct set of packages to install (see #872).

stain pushed a commit that referenced this issue Aug 21, 2018

Bump dependency on requests from 2.4.3 to 2.6.1

499e3d1

cwltool itself doesn't depend on requests 2.6.1, but CacheControl does, and having matching dependencies helps Pip find a correct set of packages to install (see #872).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip install cwltool failed with mypy-extension==0.4.0 #872

pip install cwltool failed with mypy-extension==0.4.0 #872

inutano commented Aug 17, 2018

manabuishii commented Aug 17, 2018 •

edited

Loading

sersorrel commented Aug 17, 2018

mr-c commented Aug 17, 2018

inutano commented Aug 17, 2018

sersorrel commented Aug 17, 2018

inutano commented Aug 18, 2018

inutano commented Aug 18, 2018

sersorrel commented Aug 20, 2018

mr-c commented Aug 20, 2018

mr-c commented Aug 21, 2018

pip install cwltool failed with mypy-extension==0.4.0 #872

pip install cwltool failed with mypy-extension==0.4.0 #872

Comments

inutano commented Aug 17, 2018

Expected Behavior

Actual Behavior

Full Traceback

Your Environment

manabuishii commented Aug 17, 2018 • edited Loading

sersorrel commented Aug 17, 2018

mr-c commented Aug 17, 2018

inutano commented Aug 17, 2018

sersorrel commented Aug 17, 2018

inutano commented Aug 18, 2018

inutano commented Aug 18, 2018

sersorrel commented Aug 20, 2018

mr-c commented Aug 20, 2018

mr-c commented Aug 21, 2018

manabuishii commented Aug 17, 2018 •

edited

Loading