_NamespacePath object has no attribute sort, again

[hagenw]

• Pip version: 10.0.0.dev0
• Python version: 3.5.2
• Operating system: Ubuntu 16.04

Description:

The bug described in #4216 still happens for me with the latest master version (094afef) of pip.

What I've run:

virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
pip install git+https://github.com/pypa/pip.git@master
pip install git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

[...]

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
        from setuptools.extern.six.moves import filter, map
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
        from pkg_resources.extern import VendorImporter
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2927, in <module>
        @_call_aside
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
        f(*args, **kwargs)
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2952, in _initialize_master_working_set
        add_activation_listener(lambda dist: dist.activate())
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 956, in subscribe
        callback(dist)
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2952, in <lambda>
        add_activation_listener(lambda dist: dist.activate())
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2515, in activate
        declare_namespace(pkg)
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2097, in declare_namespace
        _handle_ns(packageName, path_item)
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2047, in _handle_ns
        _rebuild_mod_path(path, packageName, module)
      File "/home/hagen/git/sfs-theory/tmp/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
        orig_path.sort(key=position_in_sys_path)
    AttributeError: '_NamespacePath' object has no attribute 'sort'

[pradyunsg]

Hi @hagenw!

Could you provide the complete output of the run causing the crash with --verbose option?

[hagenw]

I repeated it with your proposed modification:

virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
pip install git+https://github.com/pypa/pip.git@master
pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig \
    > pip-error.log

The content of pip-error.log is available at
https://gist.github.com/hagenw/9c29d62834edead374daa75bad5641f5.

If I do the installation manually with

git clone https://github.com/hagenw/sphinxcontrib-katex
cd sphinxcontrib-katex
virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
pip install git+https://github.com/pypa/pip.git@master
python setup.py install

it works without any error.

[pradyunsg]

@hagenw it seems that your setuptools version is not new.

Could you try upgrading setuptools and running the same command?

[AgarwalSaurav]

Hi,

I am having a similar issue. I get the error even when I run pip3 -V. If you can have a look at [Stack Overflow](https://stackoverflow.com/questions/48695928/issues-with-pip it would be great.)

I have tried updating and upgrading setuptools (and all standard updates).

[hagenw]

@pradyunsg: I repeated my commands and checked that I have the latest setuptools installed:

pip show setuptools
Name: setuptools
Version: 38.5.1

I'm getting exactly the same error as posted above.

[glenn-horton-smith]

Is it correct to close this bug before the patch is made in pypi/pip? The above mentioned commit is to the CarlosDutra/pip fork, but it has not been pulled into pypi/pip yet. (And when I did "pip3 --upgrade pip" today, the manual patch I had made disappeared, causing the bug to return.)
I might be confused by the formatting of this page. It says "Closed" at the end of the comment above, but "Open" at the top of this page.

[glenn-horton-smith]

Ah, I see at #5103 that it was the pull request that was closed.
Furthermore, the discussion of the pull request included the statement "IIRC, a newer version of setuptools would fix this."
For me, "pip3" reported that the setuptools requirement was satisfied by the existing installation setuptools 38.5.2. (This is after making the local patch to get pip working again) But when I told it "pip3 install --user --upgrade setuptools", it downloaded and installed setuptools 39.0.1. And after that, there is no need for the patch to pip/_vendor/pkg_resources/__init__.py. So the newer version of setuptools does seem to the trick. The higher version of setuptools just isn't being required on an upgrade to pip.

It would be great if pip would require a setuptools version high enough (>= 39.0.1?) that this problem would stay fixed on an upgrade.

[hagenw]

Does this mean the bug should be fixed by using version 39.0.1 of setuptools? For me this is not the case. I repeated the commands from my earlier comment and still get the _NamespacePath sort method error with pip master (0007825) and setuptools 39.0.1. Those commands were:

virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
pip install git+https://github.com/pypa/pip.git@master
pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig \
    > pip-error.log

[glenn-horton-smith]

tl;dr: Yes, at least it works for me.
When I do your first two steps, I have setuptools 39.0.1 and pip version 9.0.3, which is the latest release. This version of pip does not have any patch to pip/_vendor/pkg_resources/__init__.py and yet it works without the _NamespacePath issue. So the problem does not occur with the latest pip and setuptools version working together.
And it still works after I install pip from pypa/pip.get@master using your next step. It reports "Successfully installed pip-10.0.0.dev0", pip --version reports the same, and I can install pip packages, e.g., pip install sympy works just fine.

[pradyunsg]

@hagenw Looking at the Gist again, I see that the traceback is coming from setuptools but I can't tell the version. Could you update it?

If you're still having failures with newest setuptools and pip, could you post a new gist or update the previous one with the output with the newest versions of pip and setuptools?

[hagenw]

I added another file to the gist containing the error message: pip-error2.log.

I used pip 10.0.0.dev0 and setuptools 39.0.1 as reported by pip show ... and executed the following commands:

virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
pip install git+https://github.com/pypa/pip.git@master
pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig \
    > pip-error2.log

The old error message is also still available for comparison.

[pradyunsg]

Is the setuptools inside the repo of the right version? I see that 39.0.1 doesn't have the same line contents as the traceback suggests.

[hagenw]

Is there a way to test this? I made sure that tmp didn't existed before calling virtualenv and after the pip update I checked for setuptools by pip show setuptools which reported 39.0.1.

[pradyunsg]

@jaraco might have inputs here.

I'm not entirely sure what is happening here.

[pradyunsg]

@hagenw if you find the time, what's the (complete) output of the following:

Click to expand

echo "General Env"
pip --version
which python3
python3 -c "import setuptools; print(setuptools.__version__)"

virtualenv --python=python3 --no-site-packages tmp
source tmp/bin/activate
echo "Inside venv"
pip --version
python3 -c "import setuptools; print(setuptools.__version__)"
pip install --quiet git+https://github.com/pypa/pip.git@master
pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

[hagenw]

I executed the commands and pasted the output with 2&> to https://gist.github.com/hagenw/9c29d62834edead374daa75bad5641f5#file-pip-error3-log

[jaraco]

First, a summary. The issue in setuptools was corrected (more of a workaround) in 31.0.1. So if you're encountering the traceback, you should investigate where that older version of setuptools originated. If 'pip/_vendor' is in the traceback for pkg_resources, then it's pip's vendored version that is implicated.

I executed the commands and pasted the output

That is surprising and unexpected output. I'll see if I can replicate the issue.

[jaraco]

I created this Dockerfile to attempt to replicate the issue:

FROM ubuntu:xenial

RUN apt update && apt install -y python3 wget git
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python3
RUN pip install virtualenv
RUN virtualenv env
RUN env/bin/pip install --quiet git+https://github.com/pypa/pip.git@master
RUN env/bin/pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

I was unsuccessful. The docker build completed without error.

@hagenw: Perhaps you have a virtualenv that's somehow pulling in an older setuptools? What do you get for virtualenv --version? Can you replicate the issue in a Docker container?

[pfmoore]

it's pip's vendored version that is implicated

From a bit of digging, it appears to me that pip 10.0 has setuptools > 31.0.1 (master was updated to 34.3.2 on Mar 18 2017). Releases in the 9.0.x series have pkg_resources from before then. Specifically, the commit mentioned in the OP's report (094afef) has a sufficiently new version of setuptools.

FWIW, the OP is running on Ubuntu, and their unbundling of pip's vendored dependencies may be involved here (although the issue is reported against what appears to be an unmodified copy of pip from master, so I don't see how).

[hagenw]

@jaraco: I tested your docker file and it also builds without error on my machine.

virtualenv --version
15.0.1

I can also reproduce the error by using python3 -m venv tmp instead of virtualenv --python=python3 --no-site-packages tmp.

[jaraco]

@hagenw I also am able to replicate the issue if I use python -m venv, and that's because unlike virtualenv, venv doesn't install the latest setuptools. So if you use venv, you also need to upgrade setuptools. Or more concretely, this Dockerfile replicates the failure:

FROM ubuntu:xenial

RUN apt update && apt install -y python3-venv wget git
RUN python3 -m venv env
RUN env/bin/pip install --quiet git+https://github.com/pypa/pip.git@master
RUN env/bin/pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

And this one employs the fix:

FROM ubuntu:xenial

RUN apt update && apt install -y python3-venv wget git
RUN python3 -m venv env
RUN env/bin/pip install --quiet git+https://github.com/pypa/pip.git@master
RUN env/bin/pip install -U setuptools
RUN env/bin/pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

[hagenw]

Cool, I can also fix it with pip install -U setuptools for python3 -m venv tmp on my machine.

Interestingly, this can fix the bug for virtualenv --python=python3 --no-site-packages tmp as well even if it reports to have the newest version of setuptools installed.
I just have to force a reinstallation and it will work:

virtualenv --python=python3 --no-site-packages tmp
pip install git+https://github.com/pypa/pip.git@master
pip install --force -U setuptools
pip install --verbose git+https://github.com/hagenw/sphinxcontrib-katex@math_numfig

[GaganaB]

Re-installing Python worked for me! :)

[pradyunsg]

This error was a bug in an older version of setuptools which pip had vendored. Upgrading to an newer version of pip and setuptools should resolve this issue.

Closing since I don't think there's anything actionable here from pip's end.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.