Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assertion when using --no-cache-dir in 19.0 #6158

Closed
snstanton opened this issue Jan 22, 2019 · 56 comments · Fixed by #6171
Closed

assertion when using --no-cache-dir in 19.0 #6158

snstanton opened this issue Jan 22, 2019 · 56 comments · Fixed by #6171
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Milestone

Comments

@snstanton
Copy link

Environment

Running in a dockerfile.

Description

The following command works with pip 18.1 and fails with 19.0.

pip3 install --no-cache-dir --upgrade -r requirements.txt

With 19.0, it fails with the following exception:

Exception:
Traceback (most recent call last):
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Removing the --no-cache-dir flag causes the install to succeed.
requirements.txt

@Mohitsharma44
Copy link

Mohitsharma44 commented Jan 22, 2019

Same thing happening with:
Python v3.6.8
pip version 18.1
on
Ubuntu:latest image.

@snstanton what base image are you using? I'm seeing a similar issue on pip v18.1 as well

@gst
Copy link

gst commented Jan 22, 2019

I've got the exact same issue.
on my side it seems it doesn't matter which package/distribution I try to install

@dan-blanchard
Copy link

dan-blanchard commented Jan 22, 2019

I'm seeing this even without --no-cache-dir set. It happens for all packages I try to install, even if they're already installed.

  • pip version: 19.0
  • Python version: 3.6.0
  • OS: Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

I should note that in my case I'm seeing it when running pip with a combination of sudo -H and bash -l -c.

$ sudo -H bash -l -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Looking in indexes: https://pypi.org/simple, http://pypi.lan.cogtree.com/cogtree/simple/
Collecting hypothesis
  Downloading http://pypi.lan.cogtree.com/cogtree/simple/hypothesis/hypothesis-4.1.0-py3-none-any.whl (238kB)
    100% |████████████████████████████████| 245kB 120.5MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Exception:
Traceback (most recent call last):
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Running the same commands without -l on my bash -c, or without bash -l -c involved at all, it all works fine.

$ sudo -H bash -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Collecting hypothesis
  Downloading https://files.pythonhosted.org/packages/89/7b/d6206dcde963139daa03a1d85b0c3428cb3ebf2ae8de3244b14a63e22680/hypothesis-4.1.0.tar.gz (180kB)
    100% |████████████████████████████████| 184kB 33.7MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Building wheels for collected packages: hypothesis
  Building wheel for hypothesis (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/10/12/eb/4ab734432e8466d545c8501f531458845b45e8c4427d5367f9
Successfully built hypothesis
Installing collected packages: hypothesis
Successfully installed hypothesis-4.1.0

Fascinatingly, if I run the same command without sudo or bash involved at all, it still fails with the same error, so it seems like it's some weird permissions issue maybe.

@tgs
Copy link
Contributor

tgs commented Jan 22, 2019

Another workaround for some situations

For people who hit this bug because virtualenv is automatically installing the latest version of pip, you can work around it by giving virtualenv the --no-download option, or setting VIRTUALENV_NO_DOWNLOAD=1.

But be aware that this may give you a very old version of pip, depending on the last time you upgraded virtualenv.

This also works with tox: VIRTUALENV_NO_DOWNLOAD=1 tox.

@gst
Copy link

gst commented Jan 22, 2019

for what it's worth : it also fails with same error if package is already installed:

gregory.starck@canon:~/tmp$ ./venv/bin/pip install --no-cache-dir six ; echo $?
Looking in indexes: http://pypi:3141/root/ax/+simple/
Requirement already satisfied: six in ./venv/lib/python3.6/site-packages (1.12.0)
Exception:
Traceback (most recent call last):
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
2
gregory.starck@canon:~/tmp$

@cherls
Copy link

cherls commented Jan 22, 2019

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

@D0han
Copy link

D0han commented Jan 22, 2019

Issue is with failing assert, so setting env PYTHONOPTIMIZE=1 (or with parameter -O) makes this error go away.
Just tested it.
This workaround works because python optimizes code removing all asserts as one of the things.
Do not go for =2 (or -OO), as this removes docstrings and other tracebacks will appear - some code wants to operate on them.

@vinnyfuria
Copy link

It looks like someone knew this might end up being an issue (source):

        # TODO: This check fails if --no-cache-dir is set. And yet we
        #       might be able to build into the ephemeral cache, surely?
        building_is_possible = self._wheel_dir or (
            autobuilding and self.wheel_cache.cache_dir
        )
        assert building_is_possible

@burkestar
Copy link

#5884 looks like this is a related change that could have caused this?

@burkestar
Copy link

burkestar commented Jan 22, 2019

Seems like the pip maintainers should rollback the recent 19 release to address this breaking change?
19.0 release notes: https://github.com/pypa/pip/blob/master/NEWS.rst#190-2019-01-22

UPDATE: not trying to cast aspersions here, was just suggesting as one way to quickly unblock people affected by this seeing as the release had just happened. Rolling forward with a hotfix works too. I appreciate the hard work of the community that supports this mission critical tooling, and agree with the sentiments below about postmortems to learn from mistakes and prevent future issues. Meanwhile, we are doing the same internally, which means a liberal amount of hardpinning pip versions in all the places :)

@cjerdonek
Copy link
Member

cjerdonek commented Jan 22, 2019

The PR adding the TODO comment also has this comment in reply: https://github.com/pypa/pip/pull/5743/files#r215832743

Based on that comment and also the commenter above saying that passing PYTHONOPTIMIZE=1 makes the error go away, it seems like simply removing the assertion might be the correct fix (independent of the question of rolling back).

@tgs
Copy link
Contributor

tgs commented Jan 22, 2019

Yeah, when I delete that assert, packages do install fine with --no-cache-dir. In that case, it says it's Running setup.py install instead of Building wheel for sdist packages.

@matthewfeickert
Copy link
Contributor

matthewfeickert commented Jan 22, 2019

This is also happening to my projects. I can reproduce this in Docker images built FROM ubuntu:bionic and FROM centos:centos7 where I am installing Python 3 from source (here is a Gist that demonstrates a failing example for both of those Docker images in the event that it might be helpful). For the requirements.txt in the example in the Gist and

$ pip3 install --upgrade pip setuptools wheel
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages (19.0)
Requirement already up-to-date: setuptools in /usr/lib/python3.6/site-packages (40.6.3)
Requirement already up-to-date: wheel in /usr/lib/python3.6/site-packages (0.32.3)

then

$ pip3 install --upgrade --no-cache-dir -r requirements.txt

fails with

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/usr/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

but

$ pip3 install --upgrade -r requirements.txt

works fine as expected.

@asottile
Copy link
Contributor

asottile commented Jan 22, 2019

I'm particularly hitting this with tox + docker + ENV PIP_NO_CACHE_DIR=off

My workaround is to use tox-virtualenv-no-download plugin to prevent pip from auto-updating

@coderanger
Copy link

We also have --no-cache-dir in our installs inside Docker to keep the images small. Our workaround has been --cache-dir=/pipcache and then rm -rf /pipcache in the same RUN step so it never ends up in the image.

marstr added a commit to Azure/azure-cli that referenced this issue Jan 23, 2019
@zehauser
Copy link

Software development is hard and bugs like this are always going to happen. Certainly nobody should blame the pip maintainers or contributors for this incident.

However, I would suggest that this bug merits some sort of post-mortem analysis on the part of the pip team, due to the number of (missed) opportunities for this bug to have been caught before slipping into a general release. For example:

  • automated testing of core functionality like --no-cache-dir
  • pre-commit, pre-merge, or pre-release checks that flag (or prohibit) TODOs
  • a (human) pre-merge review of all unresolved review comments in the PR (Github autominimizes most review comment threads when their associated code has been changed, and as of recently lets you manually mark review comment threads as resolved)
  • changes in the release process — why not release a beta first and then wait several weeks before a general release?
  • etc

A postmortem could result in a number of helpful betterments to ensure that software that is as core to the Python project as pip doesn't ship with bugs of this magnitude in future.

@pradyunsg pradyunsg added the type: bug A confirmed bug or unintended behavior label Jan 23, 2019
@erinxocon
Copy link

I can replicate this bug. Removing --no-cache-dir fixes it. Since I don't want it in my docker image I'm using the solution @coderanger proposed. Cheers 🌈🍰🌈

@cjerdonek
Copy link
Member

Earlier in the thread I had asked if including --no-use-pep517 along with --no-cache-dir makes things work for people, but I didn't see a reply. For people that are still experiencing the option, can you try that?

@dmulter
Copy link

dmulter commented Jan 23, 2019

Adding the --no-use-pep517 option fixed the problem for me. Hope that helps narrow things down.

@Estartu
Copy link

Estartu commented Jan 24, 2019

pip 19.0.1 working for me in a virtualenv. But inside Jenkins (Shining Panda) it still fails. Adding --no-use-pep517 fixes the problem

@cjerdonek
Copy link
Member

I'm reopening since some people are still experiencing the same issue.

@cjerdonek cjerdonek reopened this Jan 24, 2019
rjw57 added a commit to uisautomation/dockerimages that referenced this issue Jan 24, 2019
pip version 19 has a bug[1] which is breaking our builds. Until it's
fixed, pin pip to a pre-19 version.

[1] pypa/pip#6158
pradyunsg referenced this issue Jan 24, 2019
When the ephemeral cache is used, the build can always occur. There is
no need to check for those.
@pradyunsg pradyunsg added C: PEP 517 impact Affected by PEP 517 processing and removed C: PEP 517 impact Affected by PEP 517 processing labels Jan 24, 2019
@rifelpet
Copy link

rifelpet commented Jan 24, 2019

I can also confirm that --no-use-pep517 fixed the issue for me after upgrading to pip 19.0.1 did not.

@hrw

This comment has been minimized.

@cjerdonek
Copy link
Member

cjerdonek commented Jan 24, 2019

At @pradyunsg's request, I've opened a new issue (#6197) specific to the AssertionError in the 19.0.1 release, as it's narrower in scope and will need new investigation. So I'm reclosing this issue.

@marcstreeter
Copy link

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

or your FROM python:3.6-alpine can change to FROM python:3.6.7-alpine

boogheta added a commit to medialab/hyphe that referenced this issue Jan 30, 2019
they are failing because pycparser has issues with --no-cache-dir
solution picked from pypa/pip#6158 (comment)
seblucas added a commit to seblucas/alpine-homeassistant that referenced this issue Jan 31, 2019
crazy-max added a commit to crazy-max/docker-firefox-syncserver that referenced this issue Feb 4, 2019
@lock
Copy link

lock bot commented May 29, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019
@lock lock bot locked as resolved and limited conversation to collaborators May 29, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.