2018.11.14 regression: pipenv fails to install editable extras #3230

digitalresistor · 2018-11-15T19:48:43Z

Issue description

[vagrant@cluster-01 src]$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
✘ Locking Failed!
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 695, in resolve_deps
    req_dir=req_dir,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 466, in actually_resolve_deps
    deps, index_lookup, markers_lookup, project, sources,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 237, in get_resolver_metadata
    req = Requirement.from_line(dep)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1072, in from_line
    r = FileRequirement.from_line(line_with_prefix)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 528, in from_line
    return cls.create(**arg_dict)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 460, in create
    setup_info = SetupInfo.from_ireq(ireq)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 391, in from_ireq
    created.get_info()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 309, in get_info
    self.get_egg_metadata()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 281, in get_egg_metadata
    metadata = get_metadata(self.setup_py.parent.as_posix(), pkg_name=self.name)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 146, in get_metadata
    for req in _deps
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3023, in parse
    req, = parse_requirements(s)
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2970, in parse_requirements
    yield Requirement(line)
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2979, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'testing'"
File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 695, in resolve_deps
    req_dir=req_dir,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 466, in actually_resolve_deps
    deps, index_lookup, markers_lookup, project, sources,
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/utils.py", line 237, in get_resolver_metadata
    req = Requirement.from_line(dep)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1072, in from_line
    r = FileRequirement.from_line(line_with_prefix)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 528, in from_line
    return cls.create(**arg_dict)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 460, in create
    setup_info = SetupInfo.from_ireq(ireq)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 391, in from_ireq
    created.get_info()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 309, in get_info
    self.get_egg_metadata()
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 281, in get_egg_metadata
    metadata = get_metadata(self.setup_py.parent.as_posix(), pkg_name=self.name)
  File "/home/vagrant/.local/lib/python2.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 146, in get_metadata
    for req in _deps
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3023, in parse
    req, = parse_requirements(s)
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2970, in parse_requirements
    yield Requirement(line)
  File "/home/vagrant/.local/share/virtualenvs/src-uiGXbgQW/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2979, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'testing'"

Downgrading to pipenv version 2018.10.13, and everything works as expected.

Expected result

Pipfile.lock is generated

Actual result

💥

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv lock

$ pipenv --support

Pipenv version: '2018.10.13'

Pipenv location: '/home/vagrant/.local/lib/python2.7/site-packages/pipenv'

Python location: '/usr/bin/python2'

Python installations found:

2.7.5: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '3.10.0-862.14.4.el7.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Sep 26 15:12:11 UTC 2018',
 'python_full_version': '2.7.5',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

HISTTIMEFORMAT
PYTHONDONTWRITEBYTECODE
LESSOPEN
SSH_CLIENT
SELINUX_USE_CURRENT_RANGE
LOGNAME
USER
PATH
HOME
LANG
TERM
SHELL
SHLVL
HISTSIZE
XDG_RUNTIME_DIR
SSH_AUTH_SOCK
SELINUX_ROLE_REQUESTED
PIP_PYTHON_PATH
XDG_SESSION_ID
_
SSH_CONNECTION
SSH_TTY
OLDPWD
HOSTNAME
SELINUX_LEVEL_REQUESTED
HISTCONTROL
PWD
CPATH
PIP_SHIMS_BASE_MODULE
MAIL
LS_COLORS

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vagrant/.local/bin:/home/vagrant/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/vagrant/src

Contents of Pipfile ('/home/vagrant/src/Pipfile'):

[[source]]
url = "https://user:[email protected]/devpi/unstable/+simple/"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
server = {path = "./server", extras = ["testing"], editable = true}
"e01afd4" = {path = "./lib", extras = ["testing"], editable = true}

[requires]
python_version = "2.7"

[pipenv]
allow_prereleases = true

The text was updated successfully, but these errors were encountered:

duplicate-issues · 2018-11-15T19:49:04Z

Hey @bertjwregeer,

We did a quick check and this issue looks very darn similar to

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

digitalresistor · 2018-11-15T19:50:40Z

@duplicate-issues: No, they are not the same or similar.

frostming · 2018-11-17T01:10:47Z

Can't replicate here, can you try installing from a public pypi source?

Or can you provide more info about the package structure, say, does it contain src folder?

digitalresistor · 2018-11-18T05:32:25Z

frostming, folder structure:

.git
.gitignore
server/
    setup.py
    src/
        server/
            __init__.py
            other.py
lib/
    setup.py
    src/
        lib/
            __init__.py
            somethingelse.py
Pipfile

The projects themselves are stock standard setup tool projects. Nothing fancy going on. The issue is that pipenv or another library is adding a second set of quotes to the extra to be installed:

pkg_resources.RequirementParseError: Invalid requirement, parse error at "'testing'"

Notice the double quotes followed by the single quotes.

It doesn't matter where I install the dependencies for the project from, private or not, this is currently broken.

frostming · 2018-11-18T08:10:03Z

@bertjwregeer Please try to prune the virtualenv pipenv --rm and run pipenv lock from a clean environment.

digitalresistor · 2018-11-18T18:06:32Z

@frostming the issue occurred when running pipenv from scratch in a new environment without a Pipfile.lock available.

If I create the Pipfile.lock using an older version of pipenv and then upgrade to the new pipenv all works without issues.

techalchemy · 2018-11-19T05:22:58Z

@frostming the nuance here is going to be that this is a local package which has a setupfile, yet we are escaping the extras for reasons which make no discernible sense. I'm going to venture a random guess that this is related to the upstream restructuring we made to requirementslib around setup.py parsing, we could easily be mishandling extras when converting them (either in setup_info or pipfile.

@bertjwregeer -- I tried to get as close as possible to your setup as I could but I can't reproduce the issue -- are your extras in your setup.py or setup.cfg or are they configured with pbr or some other tooling? We jump through some hoops to try and parse that all, so it's possible in some specific cases we are pulling in some stray quotation marks

here was my test run (using setup.cfg files):

[[sources]]
url = "https://pypi.org/simple"
name = "pypi"
verify_ssl = true

[packages]
pylint = "*"
installer = {path = "./installer", editable = true, extras = ["virtualenv"] }
vistir = { path = "./vistir", editable = true, extras = ["spinner"] }

[dev-packages]

[requires]
python_version = "2.7"

[pipenv]
allow_prereleases = true

/t/test2  pipenv install --dev
Creating a virtualenv for this project…
Pipfile: /tmp/test2/Pipfile
Using /home/hawk/.pyenv/versions/2.7.15/bin/python2 (2.7.15) to create virtualenv…
⠏ Creating virtual environment...New python executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python2
Also creating executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /home/hawk/.pyenv/versions/2.7.15/bin/python2

✔ Successfully created virtual environment! 
Virtualenv location: /home/hawk/.virtualenvs/test2-h6PFdzq5
Pipfile.lock (a7c78d) out of date, updating to (d69bb7)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (a7c78d)!
Installing dependencies from Pipfile.lock (a7c78d)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 47/47 — 00:00:09
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

digitalresistor · 2018-11-19T16:10:21Z

My extras are defined in setup.py.

xistence$ cat server/setup.py
import os
from setuptools import setup, find_packages

thisdir = os.path.abspath(os.path.dirname(__file__))
version = open(os.path.join(thisdir, 'version.txt'), 'rb').read().strip()

testing_extras = [
    'beautifulsoup4',
    'coverage',
    'cr.lib[testing]',
    'flaky',
    'mock==1.0.1',
    'pycrunch>=0.2.1',
    'pytest',
    'pytest-cov',
    'pytest-xdist',
    'randomize',
    'requests_mock',
    'moto',
    'ming',
]

setup(
    name='cr.server',
    version=version,
    description="The Backend HTTP Server",
    long_description=open(os.path.join(thisdir, 'README.md')).read(),
    install_requires=[
        'CherryPy>=11.0.0',
        'cheroot>=5.3.8',
        'PyYAML>=3.10',
        'argparse>=1.2.1',
        'pymongo>=3.4.0',
        'pytz>=2011k',
        'repoze.who>=2.1b1',
        'zope.interface>=4.0.3',
        'docopt',
        'Dozer>=0.4',
        'openpyxl',
        'python-dateutil',
        'unicodecsv',
        'requests',
        'requests-cache',
        'six',
    ],
    tests_require=testing_extras,
    extras_require={
        'testing': testing_extras,
    },
    packages=find_packages(where='src'),
    package_dir={"": "src"},
    namespace_packages=['cr'],
    include_package_data=True,
    zip_safe=False,
)

- Fixes #3230 Signed-off-by: Dan Ryan <[email protected]>

techalchemy · 2018-11-19T23:00:13Z

Awesome, that was exactly what I needed to sort this out. Thanks for the excerpt

frostming mentioned this issue Nov 16, 2018

Fix environment #3243

Merged

frostming added the Status: Needs More Information This issue does not provide enough information to take further action. label Nov 17, 2018

techalchemy mentioned this issue Nov 19, 2018

Local file requirements with extras are parsed incorrectly sarugaku/requirementslib#116

Closed

techalchemy added a commit that referenced this issue Nov 19, 2018

Fix parsing of named setup.py extras

dc89130

- Fixes #3230 Signed-off-by: Dan Ryan <[email protected]>

techalchemy mentioned this issue Nov 19, 2018

Fix parsing of named setup.py extras #3256

Merged

techalchemy added this to the November Bugfix Release milestone Nov 19, 2018

techalchemy closed this as completed in #3256 Nov 20, 2018

mschwager mentioned this issue Dec 15, 2020

Move package information to dunder attributes semgrep/semgrep#2253

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2018.11.14 regression: pipenv fails to install editable extras #3230

2018.11.14 regression: pipenv fails to install editable extras #3230

digitalresistor commented Nov 15, 2018

duplicate-issues bot commented Nov 15, 2018

digitalresistor commented Nov 15, 2018

frostming commented Nov 17, 2018

digitalresistor commented Nov 18, 2018

frostming commented Nov 18, 2018

digitalresistor commented Nov 18, 2018

techalchemy commented Nov 19, 2018

digitalresistor commented Nov 19, 2018 •

edited

Loading

techalchemy commented Nov 19, 2018

2018.11.14 regression: pipenv fails to install editable extras #3230

2018.11.14 regression: pipenv fails to install editable extras #3230

Comments

digitalresistor commented Nov 15, 2018

Issue description

Expected result

Actual result

Steps to replicate

duplicate-issues bot commented Nov 15, 2018

digitalresistor commented Nov 15, 2018

frostming commented Nov 17, 2018

digitalresistor commented Nov 18, 2018

frostming commented Nov 18, 2018

digitalresistor commented Nov 18, 2018

techalchemy commented Nov 19, 2018

digitalresistor commented Nov 19, 2018 • edited Loading

techalchemy commented Nov 19, 2018

digitalresistor commented Nov 19, 2018 •

edited

Loading