Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

[ramonamis]

• Pip version: 10.0.0
• Python version: 2.7
• Operating system: Amazon ECS-Optimized Amazon Linux AMI 2017.09.i

Description:

I am trying to install docker-py, it is a usual part of our infrastructure setup workflow and we run it for quite some time. I get the following error for some packages:

Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Everything works as expected with pip 9.0.2. Did something change? What can I do to fix this? (except pinning the version of pip to 9.0.2 or 9.0.3)

What I've run:

First installation attempt with pip 10

# /usr/local/bin/pip2 install docker-py
Collecting docker-py
  Using cached docker_py-1.10.6-py2.py3-none-any.whl
Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/site-packages (from docker-py) (1.11.0)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from docker-py) (3.5.0.1)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/site-packages (from docker-py) (1.0.22)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/site-packages (from docker-py) (0.47.0)
Collecting requests!=2.11.0,>=2.5.2 (from docker-py)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting docker-pycreds>=0.2.1 (from docker-py)
  Using cached docker_pycreds-0.2.2-py2.py3-none-any.whl
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (2018.1.18)
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,>=2.5.2->docker-py)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: chardet, requests, docker-pycreds, docker-py
  Found existing installation: chardet 2.0.1
Cannot uninstall 'chardet'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Downgraded to pip 9.0.3 and then got this:

# /usr/local/bin/pip2 install docker-py
Collecting docker-py
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading docker_py-1.10.6-py2.py3-none-any.whl (50kB)
    100% |████████████████████████████████| 51kB 4.8MB/s
Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/site-packages (from docker-py)
Collecting requests!=2.11.0,>=2.5.2 (from docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 8.2MB/s
Collecting docker-pycreds>=0.2.1 (from docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading docker_pycreds-0.2.2-py2.py3-none-any.whl
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,>=2.5.2->docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 7.7MB/s
Installing collected packages: chardet, requests, docker-pycreds, docker-py
  Found existing installation: chardet 2.0.1
    DEPRECATION: Uninstalling a distutils installed project (chardet) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling chardet-2.0.1:
      Successfully uninstalled chardet-2.0.1
  Found existing installation: requests 1.2.3
    Uninstalling requests-1.2.3:
      Successfully uninstalled requests-1.2.3
Successfully installed chardet-3.0.4 docker-py-1.10.6 docker-pycreds-0.2.2 requests-2.18.4
You are using pip version 9.0.3, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I also noticed issues with installing awscli, it complains some packages are not installed. (Installing them first fixes the issue though).

If i try to forcereinstall awscli for example, I get the same error for PyYAML:

 It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. ```

[pfmoore]

Se #4805 and #3389 for the history. Basically, pip cannot properly uninstall packages installed by "pure" distutils, because distutils doesn't record enough metadata for us to do so. We've previously removed the installation metadata, so that it looks like we did the install, but we've had to leave files behind. That causes issues. Since pip 8, we've been trying to stop doing this, because it's a cause of problems, but our initial attempt was reverted because it affected too many people at the time. With pip 10, we've finally stopped trying to uninstall distutils packages, and now we report the problem to the user as you see here.

Basically, if you (or some infrastructure you use) installed a package using distutils, you need to manage (and in particular) uninstall it "using distutils". Unfortunately distutils doesn't include an uninstall command, so "uninstall using distutils" means manually removing the package.

[ramonamis]

@pfmoore Thanks for the quick answer!

This is rather inconvenient because most of the packages come installed as dependencies and we do not manage it ourselves. Automating the removal will be interesting.

I see there is some movement to only upgrade packages, without uninstalling them first. It would be great if this would eventually happen.

We could close this issue since it is widely discussed somewhere else. Thanks!

[shubhamyedage]

Try removing package manually from "site-packages". This works perfect!

[jclinto1]

I seem to get round this problem by providing the version of pip with the command
pip install -I==9.0.3 -r requirements.txt

[wangxf1987]

I used follow steps and resolved this problem

1. Reduced version:
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package:
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip:
   pip install --upgrade pip

[JohnBDonner]

This seems to fix the issue for me: stacks-network/stacks-core#504

In the orignal case above, that'd be:

pip install docker-py --ignore-installed PyYAML

[AleJuliet]

Don't know who you are, but I love you.

[ruby-is-pretty-cool]

So @pfmoore what would you recommend for a team that automates their pip installations? We set up our server using terraform, so we automatically call pip install for smart-open and boto3 and then we have numpy, scipy, boto, sklearn, and datadog in a requirements.txt. We get the distutils error for urllib3 for multiple installs (because smart-open installed it). The installs work when I use --ignore-installed urllib3, but is there a more proper way to go about this?

Also, is there any chance that distutils will add an uninstall feature or more metadata? Have you guys talked to that team about this?

[pfmoore]

@ruby-is-pretty-cool I have no real opinion on that. You say of urllib3 "because smart-open installed it" but I don't see anything in smart-open that declares a dependency on urllib3, so I don't know what you mean by that. If smart-open is somehow installing urllib3 in a way that triggers this issue, that sounds like a smart-open problem.

Also, is there any chance that distutils will add an uninstall feature or more metadata? Have you guys talked to that team about this?

Not that I know of, and no we haven't talked to them about it. I doubt they would, as distutils doesn't really implement new features these days (but feel free to ask if you want to). The official packaging documentation doesn't recommend using distutils directly anyway, so if projects are doing so, it's really up to them to manage the implications.

[pfmoore]

I just tried installing smart-open. It depends on requests, which in turn depends on urllib3. That is installed properly by pip when I do the install in a virtualenv. Are you doing this installation in a system environment? In which case, are you seeing (and trying to upgrade) the system installed urllib3? If so, the normal advice "don't use pip on system packages" applies - use a virtualenv or if you need to install to your system environment use distro packages.

[ruby-is-pretty-cool]

I am pretty new to Python packaging, but I have a feeling that either a virtualenv or distro packages will fix our issue - I'll have to look into them more. Thank you for your help!

[pabloa]

This is insane. Installations are being broken because of some religion-like believing. if some setup.py entry of some package touches urllib3, then there is no way to tells it to ignore it. In our case, we must upgrade urllib3 because in Ubuntu 14 the installed version of pip (v1.5.x) refuses to install packages from https URLs.

The point is that nobody would upgrade "system" packages if they would not stop working. In linux kernel project this behavior is "breaking userspace" and Torvalds does not react nicely to it. It is incredible python people takes it so naturally fine.

At least you should warn people in pip messages with something like "this version of pip will refuse to work if it means to upgrade system packages, do this and do that. Use pip version 9.x if you are working with old installers or old systems." or similar.

[AddoSolutions]

I agree with @pabloa - This has been going on forever now, a vanilla install of CentOS 7 cannot install docker-compose because of this.

And yes, @JohnBDonner, that little flag just saved me a LOT of time. THANK YOU

[hanc69]

This "not our problem, someone else fix it" is to be expected from pip. The software doesn't even pretend to manage package conflicts. I got this message as part of an ansible deployment of containers to many systems.

Thank you for those of you who supplied workarounds. I've changed the ordering - "pip install docker" prior to "pip install --upgrade pip" - worked this time. Hopefully this won't cause problems (such as data corruption) in future.

[pherris]

My reading of this is that we should be using virtual environments to protect ourselves from oddness in our environments... however, I was able to get past this for now by adding the --user flag:

pip3 install -r requirements.txt --user

https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-user

[rgabo]

You could also try working with the exact version of the transient dependency that was installed by your environment / distutils as it could be compatible. In my example, Pipenv will resolve the transient PyYAML dependency of awscli==1.15.85 and apache-airflow==1.10.1 to 3.13, but the system already has 3.11 installed. Looking at the declared dependencies on my local development machine, 3.11 would be fine for both:

$ pipenv graph | egrep 'airflow|awscli|PyYAML'
apache-airflow==1.10.1
    - PyYAML [required: >=3.0, installed: 3.13]
awscli==1.15.85
  - PyYAML [required: >=3.10,<=3.13, installed: 3.13]

A simple pipenv install PyYAML==3.11 will pin PyYAML at 3.11 and make both packages happy:

$ pipenv install PyYAML==3.11
Installing PyYAML==3.11…
...
Locking [packages] dependencies…
✔ Success!

$ pipenv graph | egrep 'airflow|awscli|PyYAML'
apache-airflow==1.10.1
    - PyYAML [required: >=3.0, installed: 3.11]
awscli==1.15.85
  - PyYAML [required: >=3.10,<=3.13, installed: 3.11]

My Pipfile / Pipfile.lock subsequently installs cleanly on Ubuntu 14.04LTS using pipenv install --deploy --system.

I also checked and PyYAML==3.11 is installed through python3-yaml 3.11-3build1 and it is a direct dependency of cloud-init 18.4-0ubuntu1~16.04.2, which we extensively use as we run on EC2. Both 18.04 LTS and 18.10 come with PyYAML 3.12, only 19.04 will come with PyYAML, which happens to be the latest version as of today.

That being said, avoid system dependencies and environmental issues at all costs. Use pipenv and/or virtualenv.

[s-eswar]

I used follow steps and resolved this problem:

1. Reduced version,
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip
   pip install --upgrade pip

This worked well for me. But does upgrading pip after this give any problems to the installed packages??

[wangxf1987]

@s-eswar So far, we are not found problems about the package, but maybe one situation that if use the high version of the pip install package, then use the lower version re-install or check may be occur dependency problem. I suggest that always use lower version. For instance, pip 9.0.3.

[s-eswar]

@s-eswar So far, we are not found problem about for packages, but maybe one situation that if use the high version of pip install package, and use the lower version re-install or check may be occur dependency problem. I suggest that always use lower version. For instance, pip 9.0.3.

@wangxf1987 but for using ML libraries with the same config, don't we get errors regarding upgradation/use of lower version.

[wangxf1987]

@s-eswar I am not sure ML libraries whether prefect works under lower version. I using pip=9.0.3 works will with the latest version of the Kubernetes. It's should be check the requirements file whether need to the latest version of the pip or testing in the dev env.

[noloader]

Since pip 8, we've been trying to stop doing this, because it's a cause of problems, but our initial attempt was reverted because it affected too many people at the time. With pip 10, we've finally stopped trying to uninstall distutils packages, and now we report the problem to the user as you see here.

I believe the part "now we report the problem to the user" is called "pushing your technological debt onto someone else". I'd like to thank everyone for bring me into the fiasco. Especially since I am not a Python dev and I have no idea how to fix it. I get to wates hours on this problem now. Hooray!

[jinnabaalu]

Remove Dist Package and RUN

sudo rm -rf /usr/lib/python3/dist-packages/yaml

sudo rm -rf /usr/lib/python3/dist-packages/PyYAML-*

Removing folder from distutils works

[hemangjoshi37a]

by upgrading to newer version it should solve the problems but pip introduces problems what a irony.

[swishderzy]

The solution is to uninstall it manually, so go to.../anaconda3/lib/python3.*/site-packages/ and delete all files and folders of the package

[andrewcar]

@ramonamis sudo pip install --ignore-installed PyYAML

This successfully upgraded it for me.

[transis22]

This also worked for imutils:

pip install --upgrade imutils --ignore-installed imutils

[anancds]

I used follow steps and resolved this problem

1. Reduced version:
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package:
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip:
   pip install --upgrade pip

it works for me， Thanks!

[LittleNewton]

I used follow steps and resolved this problem

1. Reduced version:
   `pip install --upgrade --force-reinstall pip==9.0.3`

2. Tried to re-install package:
   `pip install xxx --disable-pip-version-check`

3. At last, recover the latest version for pip:
   `pip install --upgrade pip`

this is a bad solution. I cannot do anything now.

Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3-pip : Depends: python-pip-whl (= 9.0.1-2.3~ubuntu1) but 9.0.1-2.3~ubuntu1.18.04.1 is to be installed
               Recommends: python3-dev (>= 3.2) but it is not going to be installed
               Recommends: python3-setuptools but it is not going to be installed
               Recommends: python3-wheel but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
E: Could not read response to hello message from hook [ ! -f /usr/bin/snap ] || /usr/bin/snap advise-snap --from-apt 2>/dev/null || true: Success

[pradyunsg]

I'm locking this thread now. @pfmoore's initial comment covers everything here. The rest of this thread has divulged into support requests or straight up abuse directed at the maintainers.