Specifying 'extras' with remote wheel causes exception #5536

elanou · 2022-12-14T20:38:02Z

Issue description

Trying to install wheel from remote file with 'extras' specified in pipfile causes exception.

pipfile ex:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'

Expected result

Install wheel and extras

Actual result

Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 102, in init
req = REQUIREMENT.parseString(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pyparsing/core.py", line 1141, in parse_string
raise exc.with_traceback(None)
pipenv.patched.pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found '#' (at char 12), (line:1, col:13)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3101, in init
super(Requirement, self).init(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 104, in init
raise InvalidRequirement(
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 275, in parse_line
req = Requirement.from_line(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2684, in from_line
r = file_req_from_parsed_line(parsed_line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 3084, in file_req_from_parsed_line
"req": parsed_line.requirement,
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 475, in requirement
self.parse_requirement()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1081, in parse_requirement
self._requirement = init_requirement(name) # type: PackagingRequirement
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/utils.py", line 192, in init_requirement
req = Requirement.parse(name)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3148, in parse
req, = parse_requirements(s)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3094, in parse_requirements
yield Requirement(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3103, in init
raise RequirementParseError(str(e))
pipenv.patched.pip._vendor.pkg_resources.RequirementParseError: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 816, in
main()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 802, in main
_main(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 782, in _main
resolve_packages(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 730, in resolve_packages
results, resolver = resolve(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 709, in resolve
return resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 1105, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 891, in actually_resolve_deps
resolver = Resolver.create(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 460, in create
constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 219, in get_metadata
req, req_idx, markers_idx = self.parse_line(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 287, in parse_line
raise ResolutionFailure(
pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy-darwin#egg=[transformers]
✘ Locking Failed!

Steps to replicate

add below to pipfile:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'

Pipenv version: '2022.9.24'

Pipenv location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/bin/python3.11'

OS Name: 'posix'

User pip version: '22.2.2'

PEP 508 Information:

{'implementation_name': 'cpython',
'implementation_version': '3.11.0',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '22.1.0',
'platform_system': 'Darwin',
'platform_version': 'Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT '
'2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103',
'python_full_version': '3.11.0',
'python_version': '3.11',
'sys_platform': 'darwin'}

The text was updated successfully, but these errors were encountered:

matteius · 2022-12-17T18:33:18Z

Can this be rechecked with pipenv==2022.12.17?

oz123 · 2022-12-18T22:00:35Z

This still happens on the latest release.

 $ pipenv install -v
Pipfile.lock not found, creating...
Locking [packages] dependencies...
⠙ pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: 
$ pipenv --version
pipenv, version 2022.12.18.dev0
$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

[pipenv]
allow_prereleases = true

Looks like a bug in requirementslib.

oz123 · 2022-12-18T22:19:28Z

Adding some more information I found out trying to debug this:

 $ .../virtualenvs/foo-_rgbUsyh/bin/python /...pypa/pipenv/pipenv/resolver.py --pre --category packages --write /tmp/resolverskea3pd8.json --constraints-file /tmp/pipenvy9wg0apmconstraints.txt
> ...pipenv/pipenv/resolver.py(802)_main()
-> os.environ["PIP_PYTHON_PATH"] = str(sys.executable)
(Pdb) c
>.../pypa/pipenv/pipenv/utils/resolver.py(273)parse_line()
-> try:
(Pdb) c
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/resolver.py", line 738, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 1102, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 890, in actually_resolve_deps
[ResolutionFailure]:       resolver = Resolver.create(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 460, in create
[ResolutionFailure]:       constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 216, in get_metadata
[ResolutionFailure]:       req, req_idx, markers_idx = self.parse_line(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 287, in parse_line
[ResolutionFailure]:       raise ResolutionFailure(
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]

oz123 · 2022-12-19T21:07:22Z

OK. This is definitely a bug in requirementslib:

(Pdb) p line
"{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]"
(Pdb) s
--Call--
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2669)from_line()
-> @classmethod
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2673)from_line()
-> if isinstance(line, InstallRequirement):
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2675)from_line()
-> parsed_line = Line(line)
(Pdb) l
2670 	    @lru_cache()
2671 	    def from_line(cls, line):
2672 	        # type: (AnyStr) -> Requirement
2673 	        if isinstance(line, InstallRequirement):
2674 	            line = format_requirement(line)
2675 ->	        parsed_line = Line(line)

oz123 · 2022-12-19T21:27:03Z

Suspected cause of problem:

The constraints from:

spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

are written to:

{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers] -i https://pypi.org/simple'}

Blubberblub · 2022-12-20T14:44:31Z

I stumbled over the same issue when trying to install a wheel for the ray library. From my pipfile:

ray = { extras = ['default'], file = "https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl" }

Gave me this error:

ERROR: Failed to resolve requirement from line: https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#egg=ray#egg=ray[default]

Originally i tried to replicate this line for the installation through pip:

pip install "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl"

When installing the url through the command line without extras the installations works:

pipenv install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl

pipenv, version 2022.12.19

Requirements with a wheel path with extras caused a constraint line with egg name which was included twice. For example: ``` 'spacy = {file = "https://files.pythonhosted.org/packages/.../spacy-3.4.3-cp39-..._x86_64.whl", extras = ["transformers"]}' ``` Produced a constraint line like the following: ``` https://files.pythonhosted.org/.../spacy-3.4.3-..._x86_64.whl#egg=spacy#egg=spacy[transformers] ``` This line triggered an unhandled excpetion `pipenv.exceptions.ResolutionFailure`, as in #5536. The should be submitted actually to requirementslib. However, I am publishing it here so others can confirm it too. Signed-off-by: Oz Tiram <[email protected]>

oz123 · 2022-12-21T12:58:05Z

@elanou @Blubberblub I have published a branch with a commit that fixes this issue. Please see the detailed commit message, as for why this is still not merged. You can, however, test already the changes.
I will add an integration test for this here, and some unit tests in requirementslib. Once the upstream PR is merged, and vendored here, we can close this issue.
This is going to take a few more days. In the meanwhile, you can apply the patch to your version of pipenv and enjoy.

Requirements with a wheel path with extras caused a constraint line with egg name which was included twice. For example: ``` 'spacy = {file = "https://files.pythonhosted.org/packages/.../spacy-3.4.3-cp39-..._x86_64.whl", extras = ["transformers"]}' ``` Produced a constraint line like the following: ``` https://files.pythonhosted.org/.../spacy-3.4.3-..._x86_64.whl#egg=spacy#egg=spacy[transformers] ``` This line triggered an unhandled excpetion `pipenv.exceptions.ResolutionFailure`, as in pypa/pipenv#5536. Signed-off-by: Oz Tiram <[email protected]>

Fix #5536. Bump vendored requirementslib to 2.2.3 which includes the actual fix. Also, add a test case for this in pipenv.

matteius added triage Type: Possible Bug This issue describes a possible bug in pipenv. labels Dec 17, 2022

oz123 self-assigned this Dec 18, 2022

oz123 added PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Status: In Progress This item is in progress. Category: Requirement and removed Type: Possible Bug This issue describes a possible bug in pipenv. triage labels Dec 21, 2022

oz123 mentioned this issue Dec 21, 2022

requirements parsing: do not include egg name twice sarugaku/requirementslib#355

Merged

oz123 added a commit that referenced this issue Dec 28, 2022

Fix extras listing twice with direct file URL

49999bd

Fix #5536. Bump vendored requirementslib to 2.2.3 which includes the actual fix. Also, add a test case for this in pipenv.

oz123 mentioned this issue Dec 28, 2022

Fix duplicate egg name when adding extras for package install from url #5555

Merged

oz123 closed this as completed in #5555 Dec 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Specifying 'extras' with remote wheel causes exception #5536

Specifying 'extras' with remote wheel causes exception #5536

elanou commented Dec 14, 2022

matteius commented Dec 17, 2022

oz123 commented Dec 18, 2022

oz123 commented Dec 18, 2022

oz123 commented Dec 19, 2022

oz123 commented Dec 19, 2022

Blubberblub commented Dec 20, 2022 •

edited

Loading

oz123 commented Dec 21, 2022

Specifying 'extras' with remote wheel causes exception #5536

Specifying 'extras' with remote wheel causes exception #5536

Comments

elanou commented Dec 14, 2022

Issue description

Expected result

Actual result

Steps to replicate

matteius commented Dec 17, 2022

oz123 commented Dec 18, 2022

oz123 commented Dec 18, 2022

oz123 commented Dec 19, 2022

oz123 commented Dec 19, 2022

Blubberblub commented Dec 20, 2022 • edited Loading

oz123 commented Dec 21, 2022

Blubberblub commented Dec 20, 2022 •

edited

Loading