setuptools >=19.4 breaks pip install builds for two-level namespace packages

[ghost]

Originally reported by: darren_birkett (Bitbucket: darren_birkett, GitHub: Unknown)

---

Running a pip install on a local directory is now breaking with setuptools >=19.4

# pip install plugins/holland.backup.random/
Processing ./plugins/holland.backup.random
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 11, in <module>
        from setuptools.extern.six.moves import filterfalse, map
      File "/usr/local/lib/python2.7/dist-packages/setuptools/extern/__init__.py", line 1, in <module>
        from pkg_resources.extern import VendorImporter
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3130, in <module>
        @_call_aside
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3116, in _call_aside
        f(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3155, in _initialize_master_working_set
        add_activation_listener(lambda dist: dist.activate())
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 956, in subscribe
        callback(dist)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3155, in <lambda>
        add_activation_listener(lambda dist: dist.activate())
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2655, in activate
        declare_namespace(pkg)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2225, in declare_namespace
        _handle_ns(packageName, path_item)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2194, in _handle_ns
        path.sort(key=sort_key)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2192, in sort_key
        return sys_path.index(_normalize_cached(os.sep.join(parts)))
    ValueError: '/tmp/pip-l8H5uM-build' is not in list

downgrading setuptools to 19.3 fixes the issue:

# pip install setuptools==19.3
Collecting setuptools==19.3
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached setuptools-19.3-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 19.4
    Uninstalling setuptools-19.4:
      Successfully uninstalled setuptools-19.4
Successfully installed setuptools-19.3

# pip install plugins/holland.backup.random/
Processing ./plugins/holland.backup.random
Installing collected packages: holland.backup.random
  Running setup.py install for holland.backup.random
Successfully installed holland.backup.random-1.0.10

We traced it back to this commit that was included in the 19.4 release:

https://bitbucket.org/pypa/setuptools/commits/1d2431a479f1737b95a96e62446a7b726d1d603e

I don't know enough about this to know if this is an actual problem with setuptools (pkg_resources) or the way the package I'm trying to install is put together (https://github.com/holland-backup/holland)

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/491

[ghost]

Original comment by robwhess (Bitbucket: robwhess, GitHub: robwhess):

---

I'm seeing this error too. In my case, it's happening when I'm installing modules that share top level package names. For example, I already have installed a module that provides somenamespace.somesubnamespace.package. Then, when I install a module that should provide somenamespace.somesubnamespace.anotherpackage, I see this error. I don't understand setuptools enough to know what in that commit is causing the problem.

[ghost]

Original comment by darren_birkett (Bitbucket: darren_birkett, GitHub: Unknown):

---

@robwhess yes, my package is organised the same way - a top level package, then some plugins that are under the same namespace. We have had to pin setuptools==19.3 for now

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Using holland as an example, I'm not encountering the reported failure, but am encountering another failure.

$ cd issue491/
$ hg clone gh://holland-backup/holland
destination directory: holland
importing git objects into hg
updating to branch default                                                                        
413 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd holland/
$ python -m pip install --root out
You must give at least one requirement to install (see "pip help install")
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python -m pip install . --root out
Processing /Users/jaraco/Dropbox/code/main/setuptools/issue491/holland
Installing collected packages: holland
  Running setup.py install for holland
Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/commands/install.py", line 311, in run
    root=options.root_path,
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 646, in install
    **kwargs
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_install.py", line 896, in install
    with open(inst_files_path, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/holland-1.0.11-py3.5.egg-info/installed-files.txt'
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python -m pip install -U pip
Collecting pip
  Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 541kB/s 
Installing collected packages: pip
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
Successfully installed pip-8.0.2
$ python -m pip install . --root out
Processing /Users/jaraco/Dropbox/code/main/setuptools/issue491/holland
Installing collected packages: holland
  Running setup.py install for holland ... done
Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 731, in install
    **kwargs
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_install.py", line 934, in install
    with open(inst_files_path, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/holland-1.0.11-py3.5.egg-info/installed-files.txt'

I get the same error when running with setuptools 19.3, so there seems to be something with the holland package that's broken. In any case, it doesn't provide a reproducible failure that can be used to track down the issue.

I'll follow robwhess' lead and see if I can reproduce it.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I was able to replicate the failure with a minimal package:

$ hg clone gh://jaraco/setuptools-issue491 issue491
importing git objects into hg
updating to branch default
8 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd issue491/
$ python -m venv env
$ env/bin/python -m easy_install --version
setuptools 18.2
$ env/bin/python -m pip install -U pip setuptools
Collecting pip
  Using cached pip-8.0.2-py2.py3-none-any.whl
Collecting setuptools
  Downloading setuptools-19.6.1-py2.py3-none-any.whl (472kB)
    100% |████████████████████████████████| 475kB 1.2MB/s 
Installing collected packages: pip, setuptools
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
  Found existing installation: setuptools 18.2
    Uninstalling setuptools-18.2:
      Successfully uninstalled setuptools-18.2
Successfully installed pip-8.0.2 setuptools-19.6.1
$ env/bin/pip install myns.sub.pkgA
Collecting myns.sub.pkgA
  Could not find a version that satisfies the requirement myns.sub.pkgA (from versions: )
No matching distribution found for myns.sub.pkgA
$ env/bin/pip install ./myns.sub.pkgA
Processing ./myns.sub.pkgA
Installing collected packages: myns.sub.pkgA
  Running setup.py install for myns.sub.pkgA ... done
Successfully installed myns.sub.pkgA-1.0
$ env/bin/pip install ./myns.sub.pkgB
Processing ./myns.sub.pkgB
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File ".../issue491/env/lib/python3.5/site-packages/setuptools/__init__.py", line 11, in <module>
        from setuptools.extern.six.moves import filterfalse, map
      File ".../issue491/env/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
        from pkg_resources.extern import VendorImporter
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3131, in <module>
        @_call_aside
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3117, in _call_aside
        f(*args, **kwargs)
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3156, in _initialize_master_working_set
        add_activation_listener(lambda dist: dist.activate())
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 954, in subscribe
        callback(dist)
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3156, in <lambda>
        add_activation_listener(lambda dist: dist.activate())
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2656, in activate
        declare_namespace(pkg)
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2226, in declare_namespace
        _handle_ns(packageName, path_item)
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2195, in _handle_ns
        path.sort(key=sort_key)
      File ".../issue491/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2193, in sort_key
        return sys_path.index(_normalize_cached(os.sep.join(parts)))
    ValueError: '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-fgpafpq_-build' is not in list

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-fgpafpq_-build

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The issue does seem to only be encountered when pip is used to install the packages. Installing the packages using easy_install works just fine.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Oddly, I don't encounter the issue in this scenario (where I was attempting to use my local build of setuptools to troubleshoot).

$ python -m venv env
$ env/bin/pip uninstall setuptools
Uninstalling setuptools-18.2:
...
Proceed (y/n)? y
  Successfully uninstalled setuptools-18.2
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ env/bin/python -m pip install -e ~/m/setuptools
Obtaining file:///Users/jaraco/m/setuptools
setuptools must be installed to install from a source distribution
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ env/bin/python ~/m/setuptools/setup.py develop 
running develop
running egg_info
writing entry points to setuptools.egg-info/entry_points.txt
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing top-level names to setuptools.egg-info/top_level.txt
writing requirements to setuptools.egg-info/requires.txt
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'setuptools/_vendor'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
running build_ext
Creating /Users/jaraco/Dropbox/code/main/setuptools/issue491/env/lib/python3.5/site-packages/setuptools.egg-link (link to .)
Adding setuptools 19.6.2.dev0 to easy-install.pth file
Installing easy_install-3.5 script to /Users/jaraco/Dropbox/code/main/setuptools/issue491/env/bin
Installing easy_install script to /Users/jaraco/Dropbox/code/main/setuptools/issue491/env/bin

Installed /Users/jaraco/Dropbox/code/main/setuptools
Processing dependencies for setuptools==19.6.2.dev0
Finished processing dependencies for setuptools==19.6.2.dev0
$ env/bin/python -m pip install ./myns.sub.pkgA
Processing ./myns.sub.pkgA
Installing collected packages: myns.sub.pkgA
  Running setup.py install for myns.sub.pkgA
Successfully installed myns.sub.pkgA-1.0
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ env/bin/python -m pip install ./myns.sub.pkgB
Processing ./myns.sub.pkgB
Installing collected packages: myns.sub.pkgB
  Running setup.py install for myns.sub.pkgB
Successfully installed myns.sub.pkgB-1.0
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I suspect the difference is the pip version, so pip may also be implicated in the issue. I've got to break for now, so I welcome others to investigate.

[ghost]

Original comment by stonebig (Bitbucket: stonebig, GitHub: stonebig):

---

there are other odd behaviours currently reported on pip project that may be related, with a different angle. A Clueless suggestion would be to check if your issue is still there when using "--no-cache-dir" in pip options

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

So... after struggling to once again create the error, I discovered that it's important to start from a clean clone of the setuptools-issue491 repository. I'm using hg, so a call to hg purge --all resets the state, allowing the failure to recur. If an egg-info command succeeds, then subsequent calls to install myns.sub.pkgB don't fail.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've confirmed the failure can be elicited even with pip 7.1.2 and even with the --no-cache-dir directive.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've done some debugging and here's what I've found:

$ env/bin/python -m pip install -vvv ./myns.sub.pkgB
Processing ./myns.sub.pkgB
  Running setup.py (path:/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ms4aogyy-build/setup.py) egg_info for package from file:///Users/jaraco/Dropbox/code/main/setuptools/issue491/myns.sub.pkgB
    Running command python setup.py egg_info
    > /Users/jaraco/Dropbox/code/main/setuptools/pkg_resources/__init__.py(2191)_handle_ns()
    -> def sort_key(p):
n
    (Pdb) > /Users/jaraco/Dropbox/code/main/setuptools/pkg_resources/__init__.py(2196)_handle_ns()
    -> path.sort(key=sort_key)
sort_key(path[0])
    (Pdb) 5
path
    (Pdb) ['/Users/jaraco/Dropbox/code/main/setuptools/issue491/env/lib/python3.5/site-packages/myns', 'myns']
sys_path
    (Pdb) ['', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python35.zip', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload', '/Users/jaraco/Dropbox/code/main/setuptools/issue491/env/lib/python3.5/site-packages', '/Users/jaraco/Dropbox/code/main/setuptools']
os.getcwd()
    (Pdb) '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ms4aogyy-build'
os.listdir('.')
    (Pdb) ['myns', 'pip-delete-this-directory.txt', 'pip-egg-info', 'setup.py']
q

Note that the (Pdb) appears late in the output due to line-buffering of the output by pip.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Normalize all paths, not excluding ''. Fixes #491.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Released as 19.6.2

[ghost]

Original comment by stonebig (Bitbucket: stonebig, GitHub: stonebig):

---

super !

[ghost]

Original comment by darren_birkett (Bitbucket: darren_birkett, GitHub: Unknown):

---

Thanks!

[ghost]

Original comment by robwhess (Bitbucket: robwhess, GitHub: robwhess):

---

Awesome. Confirming this works for me. Thanks for the quick work @jaraco!