Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip does not ignore yanked version of package in unqualified installation when it's the only non-prerelease choice #8262

Open
pganssle opened this issue May 18, 2020 · 26 comments
Labels
C: finder PackageFinder and index related code PEP implementation Involves some PEP state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior type: enhancement Improvements to functionality

Comments

@pganssle
Copy link
Member

Environment

  • pip version: 20.1
  • Python version: 3.8.2
  • OS: Arch Linux (using pyenv + virtualenv)

Description

If you have a package named mypkg with the following versions:

  • 1.0 (Yanked)
  • 2.0rc0

Running pip install mypkg will install 1.0 rather than 2.0rc0.

Expected behavior
I would have expected 2.0rc0 to be installed, since that is what would happen if mypkg only published version 2.0rc0. This seems to be an edge case not super well-clarified in PEP 592, which states:

An installer MUST ignore yanked releases, if the selection constraints can be satisfied with a non-yanked version, and MAY refuse to use a yanked release even if it means that the request cannot be satisfied at all. An implementation SHOULD choose a policy that follows the spirit of the intention above, and that prevents "new" dependencies on yanked releases/files.

I believe 2.0rc0 satisfies the constraints specified if you do not consider "didn't include --pre" to be a constraint, so depending on whether --pre or its absence a "constraint", this may be a violation of the spec. I think the fact that pip install will take release candidates if nothing else matches argues in favor of it not being considered a "constraint".

More clearly, though, it does not fit one of the two suggested implementation strategies:

  1. Yanked files are always ignored, unless they are the only file that matches a version specifier that "pins" to an exact version using either == (without any modifiers that make it a range, such as .*) or ===. Matching this version specifier should otherwise be done as per PEP 440 for things like local versions, zero padding, etc.
  2. Yanked files are always ignored, unless they are the only file that matches what a lock file (such as Pipfile.lock or poetry.lock) specifies to be installed. In this case, a yanked file SHOULD not be used when creating or updating a lock file from some input file or command.

Regardless of "spec lawyering", from a user experience point of view, I am personally in favor of pip refusing to install yanked versions at all without an exact version pin (option 1). Right now it installs them but with a warning; I think the warning could be turned into an error that lists the yanked versions, and end users who want to work around the "can't find a matching version" can add an exact version pin (though admittedly in some circumstances this can have perverse consequences if a package yanks a version because there's something wrong with it and a consumer pins to the exact version as a workaround, thus preventing an update to a later working, non-yanked version).

How to Reproduce

At the moment the tzdata package has only yanked and release candidate packages on PyPI (and only release candidate packages on Test PyPI). Until a proper release is made of tzdata, you can reproduce it with:

  • pip install tzdata: To see the bug
  • pip install --index-url https://test.pypi.org/simple/ tzdata: To see pip's behavior when only release candidates are available.
@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label May 18, 2020
@pganssle pganssle changed the title pip does not ignore unqualified package when it's the only non-release candidate choice pip does not ignore yanked version of package in unqualified installation when it's the only non-release candidate choice May 18, 2020
@pganssle pganssle changed the title pip does not ignore yanked version of package in unqualified installation when it's the only non-release candidate choice pip does not ignore yanked version of package in unqualified installation when it's the only non-prerelease choice May 18, 2020
@pradyunsg pradyunsg added C: finder PackageFinder and index related code state: needs discussion This needs some more discussion state: needs eyes Needs a maintainer/triager to take a closer look type: enhancement Improvements to functionality labels May 19, 2020
@triage-new-issues triage-new-issues bot removed S: needs triage Issues/PRs that need to be triaged labels May 19, 2020
@hroncok
Copy link
Contributor

hroncok commented Jun 22, 2020

Also note that if you have:

  • version 1 (yanked), otherwise installable on Python 3.7
  • version 2 (not yanked), has Requires: Python >=3.8

When you install the package via name (no version specified) from Python 3.7, it installs version 1.

Real life example:

https://pypi.org/project/ferrypick/#history

(__venv37__) [tmp]$ python --version
Python 3.7.7
(__venv37__) [tmp]$ pip install ferrypick
WARNING: The candidate selected for download or install is a yanked version: 'ferrypick' candidate (version 0.1.dev1 at https://files.pythonhosted.org/packages/b5/0e/04a44db1f3ce963f489f8582b92471ff60ef3c14bd90b4a5913e8e80aab1/ferrypick-0.1.dev1-py3-none-any.whl#sha256=110adcd8cf71f35c9926388e7b380103e1f22c2ebcdc8bd9acca16e405193782 (from https://pypi.org/simple/ferrypick/))
Reason for being yanked: This release is entirely broken, and it also doesn't block installation from Python version not supported by the later versions. Hence users of e.g. Python 3.7 would always get this broken version without a reasonable information about ferrypick requiring Python 3.8+.
Collecting ferrypick
  Using cached ferrypick-0.1.dev1-py3-none-any.whl (5.3 kB)
Installing collected packages: ferrypick
Successfully installed ferrypick-0.1.dev1

@pradyunsg
Copy link
Member

pradyunsg commented Aug 30, 2020

Definitely looks like a bug to me! PRs welcome to fix this one. :)

@pradyunsg pradyunsg added the state: awaiting PR Feature discussed, PR is needed label Aug 30, 2020
@joaomarccos
Copy link

joaomarccos commented Sep 9, 2020

Similar problem here.. When using more than one index and when the package candidate has yanked versions.. the pip does not ignore the yanked version even there is a version that matches a version specifier. You can simulate installing pyrsistent>=0.14.0, but before ensure you setup some another repository as proxy to pypi like Nexus.

@Jackenmen
Copy link
Contributor

@pradyunsg do you also consider this part from the issue description a bug?:

More clearly, though, it does not fit one of the two suggested implementation strategies:

  1. Yanked files are always ignored, unless they are the only file that matches a version specifier that "pins" to an exact version using either == (without any modifiers that make it a range, such as .*) or ===. Matching this version specifier should otherwise be done as per PEP 440 for things like local versions, zero padding, etc.
  2. Yanked files are always ignored, unless they are the only file that matches what a lock file (such as Pipfile.lock or poetry.lock) specifies to be installed. In this case, a yanked file SHOULD not be used when creating or updating a lock file from some input file or command.

Regardless of "spec lawyering", from a user experience point of view, I am personally in favor of pip refusing to install yanked versions at all without an exact version pin (option 1). Right now it installs them but with a warning

As in, would it be accepted if the PR made a change so that pip refuses to install the package if it's installed without version specifier (i.e. pip install -U package-name) per the suggested approach from the PEP 592?

To give a concrete example, I maintain a Red-DiscordBot on PyPI which does not have any non-yanked version supporting Python 3.9 but it does have yanked versions that did not have an upper-bound set. Currently, this allows the user to run python3.9 -m pip install -U Red-DiscordBot and successfully install the package (with a warning from pip about the selected candidate version being yanked) but what I would expect (and potentially want to make a PR for) is for it to refuse to install the package per the "suggested approach" from the PEP.

@pradyunsg
Copy link
Member

Yes, exactly that. In this example, Python 3.7 should fail with "nope, not got anything to install".

@dlukyanov
Copy link

another case
Dockerfile:

FROM registry.access.redhat.com/ubi8/ubi

RUN yum install -y \
        python38 \
        python38-pip \
        python38-devel \
        gcc-c++

# pdoc3 depends on setuptools-scm 
# current version of setuptools-scm 6.2.0 with label=yanked
# and pdoc3 uses the latest even there are stable versions present
RUN pip3 install pdoc3==0.10.0

ENTRYPOINT /bin/bash

cmd to build and fail:

docker build -t test-pdoc3 -f Dockerfile .

Error - note the setuptools_scm-6.2.0 version:

Collecting pdoc3==0.10.0
  Downloading https://files.pythonhosted.org/packages/95/be/69267c988fb7236cd60c452a4e7fb9a7991729476db490b634a07e7dfcdf/pdoc3-0.10.0.tar.gz (86kB)
      ...
      File "/tmp/pip-install-4trtluv9/pdoc3/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'

setuptools-scm has version history:

6.2.0 yanked 
6.1.1 yanked 
6.1.0 yanked 
6.1.0.dev0 pre-release + yanked 
6.0.1 stable
...

and pip takes 6.2.0...

https://pypi.org/project/setuptools-scm/6.2.0/

@pradyunsg
Copy link
Member

What version of pip is in that image @dlukyanov?

@RonnyPfannschmidt
Copy link
Contributor

"pip" is not involved in the download of setuptools_scm, thats setuptools/easy_install
please update pip and setuptools, the distribution versions are severely outdated and your build practically worked by accident,

the next release of setuptools_scm will warn better for such situations and suggest the fixes such as also installing python-setuptools_scm and/or other means of preinstalling

@notatallshaw
Copy link
Member

notatallshaw commented Sep 1, 2021

@dlukyanov Also as you installed pip from Redhat UBI then Redhat are specifically providing commercial support for the packages they distribute and you should reach out to their support. They are responsible for providing bug fixed versions of the packages they are distributing.

Though I suspect you will find that Redhat only supports installing third party packages via yum, and pip is probably provided for install of internal repositories and probably only ever using the the user flag.

@hroncok
Copy link
Contributor

hroncok commented Sep 1, 2021

Red Hat engineer here. If we get a customer report saying that our pip happily installs yanked versions, we will definitively do our best to fix that.

@hroncok
Copy link
Contributor

hroncok commented Sep 1, 2021

@notatallshaw
Copy link
Member

In fact: https://bugzilla.redhat.com/show_bug.cgi?id=2000135

Thanks, just in case you don't already know a major fix for yanked packages was implemented in pip 20.3.2: #9226 It looks like the pip that the user installs from redhat is 19.3.1?

The reason it became such an issue from 20.3 is that the new Resolver was turned on by default which backtracks on packages until a valid set of compatible requirements are found. Which inherently trades performance for correctness, and exposes users to potentially downloading a lot of package versions which is where the "yanked" issue came up.

@dlukyanov
Copy link

here the minimized dockerfile:

FROM registry.access.redhat.com/ubi8/ubi
RUN yum install -y python38 python38-pip
ENTRYPOINT /bin/bash

start docker, and run in console:

# python3 --version
Python 3.8.6

# pip3 install pip --upgrade
Successfully installed pip-21.2.4

# python3 -m pip --version
pip 21.2.4 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)

# pip3 install pdoc3==0.10.0
Collecting pdoc3==0.10.0
  ERROR: Command errored out with exit status 1:
  ...
  File "/tmp/pip-install-lu30xwb5/pdoc3_95c8ac/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'

@hroncok
Copy link
Contributor

hroncok commented Sep 1, 2021

Does it make any difference if you replace pip3 install with python3 -m pip install? In other words, does python3 -m pip --version give the same output as pip3 --version?

@pfmoore
Copy link
Member

pfmoore commented Sep 1, 2021

/tmp/pip-install-lu30xwb5/pdoc3_95c8ac/.eggs/setuptools_scm-6.2.0-py3.8.egg

Pip wouldn't install setuptools_scm as an egg like that. Looks like it's using easy_install, so maybe an old version of setuptools?

Also, pdoc3 doesn't have a pyproject.toml, so it's probably going through the "legacy" install mechanism that does setup.py install, which again will trigger the use of easy_install, I believe.

@Jackenmen
Copy link
Contributor

From the little time I put into trying to fix this issue, I found that the problematic part to me was that filtering of pre-releases happens in a completely different place than checking for yanked releases - the first is done by the indexer and the second is done by the resolver. I didn't really know how to proceed from there without the change potentially affecting a lot of unrelated code.

Before realizing that, I just went for a simple change in the resolver logic that does in fact make it reject the yanked versions but it also completely ignores pre-releases in the situations like the one in the original issue description. I suppose that a better solution might be to completely get rid of the yanking logic in the resolver(s) and just handle it all in the indexer but I haven't really tried looking into how achievable that would be. Indexer would probably have to have its own filtering logic rather than relying on packaging's Specifier.filter() as it does now.

Anyway, I just wanted to share a bit of information in case it's of any help to a potential future contributor but if it's not, that's fine too 😄

@dlukyanov
Copy link

dlukyanov commented Sep 1, 2021

Does it make any difference if you replace pip3 install with python3 -m pip install? In other words, does python3 -m pip --version give the same output as pip3 --version?

no diff - there is only one version, i just played with different options

@pradyunsg
Copy link
Member

pradyunsg commented Sep 1, 2021

@dlukyanov Could you post the complete output of the pip install run in a GitHub gist? The … in your posts are hiding vital debugging information that would make it easier for everyone else to see what is happening without spinning up a Docker container (eg: I don’t have Docker installed on any of my personal machines).

@dlukyanov
Copy link

# pip3 install pdoc3==0.10.0
Collecting pdoc3==0.10.0
  Downloading pdoc3-0.10.0.tar.gz (86 kB)
     |████████████████████████████████| 86 kB 1.3 MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.8 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py'"'"'; __file__='"'"'/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_42ow1r0
         cwd: /tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/
    Complete output (21 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py", line 15, in <module>
        setup(
      File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib64/python3.8/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 446, in __init__
        _Distribution.__init__(self, {
      File "/usr/lib64/python3.8/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 735, in finalize_options
        ep.load()(self, ep.name, value)
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/integration.py", line 29, in version_keyword
        dist.metadata.version = _get_version(config)
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 188, in _get_version
        dump_version(
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/95/be/69267c988fb7236cd60c452a4e7fb9a7991729476db490b634a07e7dfcdf/pdoc3-0.10.0.tar.gz#sha256=5f22e7bcb969006738e1aa4219c75a32f34c2d62d46dc9d2fb2d3e0b0287e4b7 (from https://pypi.org/simple/pdoc3/) (requires-python:>= 3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pdoc3==0.10.0 (from versions: 0.3.11, 0.3.12, 0.3.13, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.9.0, 0.9.1, 0.9.2, 0.10.0)
ERROR: No matching distribution found for pdoc3==0.10.0

@uranusjr
Copy link
Member

uranusjr commented Sep 2, 2021

Your issue does not look related to pip at all.

@dlukyanov
Copy link

Your issue does not look related to pip at all.

why pip takes setuptools_scm-6.2.0 that is yanked?

@hroncok
Copy link
Contributor

hroncok commented Sep 2, 2021

It looks like easy_install does that, not pip.

@RonnyPfannschmidt
Copy link
Contributor

It's a long standing setuptools issue (it can't self replace with a required version)

The pending setuptools_scm 6.3.0 release will support running on outdated setuptools again

Release will happen within the next hour

@uranusjr
Copy link
Member

uranusjr commented Nov 6, 2021

I think #10625 fixes this, but someone should add a test case to make sure.

@befeleme
Copy link

I just ran across the "second" part of the original issue here. With pip 22.1.2 and a mypkg having those releases:

1.0.0a1 pre-release
0.1.0 yanked

I would expect pip to install the pre-release 1.0.0a1 version when invoked with pip install mypkg
Instead, pip errors that it can't find any matching distribution for mypkg

When I try to install package that has only pre-releases, no such problem occurs. Pip installs the latest pre-release.

I would expect pip to behave the same way in both cases, ergo that it would install the pre-release.

@ddelange
Copy link
Contributor

Hi 👋 There's a related issue / test case: #11745

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: finder PackageFinder and index related code PEP implementation Involves some PEP state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior type: enhancement Improvements to functionality
Projects
None yet
Development

No branches or pull requests