How to bootstrap setuptools from source

[adamjstewart]

Hi, I'm a developer for the Spack package manager. We've had a setuptools package that has worked fine for a while now, and the vast majority of our Python packages depend on it. However, I went to update to the latest version of setuptools and noticed that it now depends on six, packaging, and appdirs. Unfortunately all 3 of these packages also depend on setuptools. six and packaging can fall back on distutils.core if necessary, but the setup.py for appdirs has a hard setuptools import.

It seems to me like it is no longer possible to build setuptools from source. Is this correct??

[adamjstewart]

An easy solution to this problem would be to replace:

from setuptools import setup

with:

try:                                                                            
    from setuptools import setup                                                
except ImportError:                                                             
    from distutils.core import setup   

in the appdirs setup.py. That's what six and packaging do. If you agree, I can reach out to the appdirs devs and see what they think.

[adamjstewart]

Ok, I hacked our package manager so that it can still install setuptools. https://github.com/LLNL/spack/pull/3198 contains all of the changes necessary to build setuptools from source now that the dependencies are no longer vendored. I understand the frustration that comes with having to maintain vendors of all of your dependencies; I just hope that you don't add any dependencies that cannot be built with distutils.core instead of setuptools.

[jaraco]

Can you use pip to install the dependencies from wheels (which don't require setuptools to install)? That's the recommended procedure. Or can your package manager install the wheels?

[adamjstewart]

Unfortunately, a package manager that requires another package manager kind of defeats the purpose, don't you think? It looks like other developers have already contributed patches to fix appdirs and pyparsing so that they can be built without setuptools. Assuming the developers approve those patches, we should be good for now.

[jaraco]

Yes, I sort of see that. I worry this approach may conflict with the long-term strategy for setuptools, which is that it might require arbitrary dependencies, some of which require setuptools. In fact, an early goal is to move pkg_resources into its own packages, and that package would probably require setuptools to install. Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

[adamjstewart]

Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

In general, yes, there is a reason. Our package manager is designed to install software in the world of HPC, where you may need to use exotic compilers like xlc to install something on a bluegene-q or cray supercomputer. Since we need to support hundreds of operating systems and dozens of compilers, we haven't spent much time on supporting binary package installation like Python's wheels or Homebrew's taps. We have plans to do so in the future, but it has been low priority. Obviously, the compiler/OS doesn't matter much for non-compiled packages like setuptools, but the mechanism would have to be the same.

We have dealt with circular dependencies before. For example, pkg-config depends on glib and vice versa. Luckily the developers realized this, and pkg-config comes with it's own internal copy of glib, much like setuptools used to come with its dependencies. This can be annoying since we end up having to add the same patches for glib to both packages, but it prevents a circular dependency which is nice.

We could theoretically vendor the dependencies ourselves. That seems like the easiest solution to me if we ever run into a setuptools dependency where distutils.core is insufficient to handle the installation. I see a lot of packages that use distutils.core as a fallback. Out of curiousity, how common is it to find a package that uses features of setuptools that are not available in distutils.core?

[jaraco]

I'd say it's fairly common. And some packagers might be depending on those features and not realizing it, because pip will pre-import setuptools and setuptools monkey-patches distutils, so even a package that only imports distutils.core will be installed with all the setuptools hooks when installed under pip. Of course, this all should change when PEP 518 lands and makes it possible to declare build dependencies and thus enable dropping the implicit invocation of setuptools.

[adamjstewart]

@tgamblin @svenevs may also want to get in on this discussion.

[svenevs]

Muahahahaha! @adamjstewart I hope this discussion pushes spack over the edge and you ditch the efforts of replicating PyPi. Things like numpy and scipy -- totally, keep them, if people pip install them then they are at fault (spack has never been easy, you can make minimal assumptions).

For things like ipdb or sphinx, there's literally no reason for spack to be providing those.

The below may not be appropriate for this thread, and some of it gets convoluted quickly. Please disregard that which you feel is not appropriate here, apologies in advance!

@jaraco is it correct that the general expectation will be

1. A user installs python (aka their OS, which came with python in most cases)
2. setuptools will never be packaged directly into this (the whole distutils split -- different conversation), but with the changes in Drop support for self upgrade/installation #581 are now expected to install a pre-built version.
3. This pre-built version should be independent of anything managed by pip in the ideal case.
   • AKA frown upon ye who pip install setuptools?

Is this correct, or should pip be used to install setuptools as the preferred approach on systems that have it available in both their native package manager and they received pip out of the box? The conversation got a little confusing on the other thread, setuptools is fundamental to so many things on pip, it seems reasonable to just assume it exists after a certain point.

Edit: I was browsing related issues, things seem clearer now. When PEP 518 is standardized, the pyproject.toml is the thing spack should ultimately be checking for setuptools? e.g. when pkg_resources splits, it will then be listed in the .toml for setuptools?

Do you happen to know if there is a "safe" way of configuring pip to not install / touch anything underneath the site-packages directory? E.g. only supporting virtualenv or ~/.pip? Adam and I had both looked into doing this, but it sounds like you are saying even if we found a way to prevent pip from behaving this way, that might cause problems (the "monkeypatching" business).

[tgamblin]

@jaraco: how do the plans for setuptools affect efforts by OS distro folks to create reproducible builds (podcast) of packages? It seems like these changes will make it very hard to validate a Python package entirely from source, and there are good reasons to make that possible.

Even if you have build dependencies and can install the build deps from wheels, that is still a binary in the chain that someone has to trust. So I worry a bit about this direction. I suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools (adamjstewart is right that we don't have binary installs yet, but they're not that hard to add). But I'd rather reproduce from source, and I suspect other distros would too.

@svenevs: other than reproducibility, the main issue with relying on pip in spack is that we need to be able to mirror builds on airgapped networks. Spack traverses a dependency DAG, downloads all the sources, and lets you build that on networks that aren't connected to the internet. If we rely on pip, we can't do that.

[adamjstewart]

In general, requiring yourself as a dependency to bootstrap yourself leads to a lot of headaches. For example, we are having a lot of nightmares trying to package openjdk/icedtea right now. If you need a java compiler to build a java compiler, how do you install the first java compiler? C/C++ compilers generally come with the operating system, but I don't think java compilers do.

[jaraco]

suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools

I wouldn't recommend using older versions of setuptools. That's an unsustainable approach in the long run.

You could rely on trusted wheels, or you could even vendor your own bootstrapping logic. That is, you could write your own routine that resolves the setuptools build dependencies (or hard-codes them), builds them into a viable form for import, and injects them onto sys.path before building setuptools.

Hmm. This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies. I'll give that a go.

[jaraco]

I've drafted a new release in the feature/bundle-build-deps branch and released b1. Install it with pip install -i https://devpi.net/jaraco/dev/+simple/ setuptools==34.3.0b1 or grab the sdist and test it in your build environment.

I notice that this change won't affect the most common use-case, that of pip installing setuptools from source when dependencies aren't met. It fails because the setuptools shim is invoked before setuptools has a chance to inject its build dependencies.

[adamjstewart]

This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies.

Can you elaborate on the distinction between the two options?

[jaraco]

Prior to Setuptools 34, setuptools would vendor its dependencies, including them as hacked-in copies of the dependencies. Thus import pkg_resources.extern.six would give the vendored version of six if present as pkg_resources._vendor.six or would fall back to the top-level six otherwise. It would do this even when installed. In this model, the dependencies were undeclared (nothing in install_requires).

With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

[jakirkham]

To summarize, this would allow us to build setuptools from an sdist on PyPI without other dependencies, correct?

[tgamblin]

With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

@jaraco: Ok, so if understand this correctly, we could then declare setuptools to have no (circular) build dependencies, and it would install fine from source. That is good. Then we could use the installed setuptools to install other packages as needed.

One question: are any of the build dependencies also run dependencies? If they are, it seems like we'll still need to make sure we install the bundled run dependencies along with setuptools when bootstrapping, or the installed setuptools won't run.

[jaraco]

are any of the build dependencies also run dependencies?

Yes. At the moment, the build dependencies and install dependencies are identical. So you would need those installed to invoke setuptools... which I can see is problemmatic if you then want to install the setuptools dependencies from source and they require setuptools.

I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them, but I don't recommend that.

I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

[adamjstewart]

I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them

Won't work on air-gapped clusters that have no connection to the outside world. We've been very careful about making sure that all of our packages can be installed without an internet connection.

[tgamblin]

I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

Ok, so I guess I have two questions:

1. Could we have an option to install the bundled dependencies along with setuptools? That would get us started.

2. Is that not basically the same as vendoring the dependencies? If so, what was so bad about vendoring, especially since setuptools is effectively a leaf in every DAG? I'm trying to figure out what problem is being solved by vendoring (feel free to point me at some other discussion...)

[dstufft]

It's different than just vendoring because it only takes place at build time, thus it allows people to say, upgrade the packaging that setuptools is using without having to also upgrade setuptools itself.

[tgamblin]

@dstufft: ok, so if we can get the bundled dependencies installed from source initially, so that we can use setuptools to install the other packages from source, does that fit the model, assuming the installed setuptools will use the newer versions when it finds them?

[reinout]

If I'm not too overenthusiastic... "when will it be released?" (I don't see it on pypi yet).

[jaraco]

Well, it should have been released automatically when I tagged the commit. I'll have to investigate why.

[jaraco]

Aah, there was an error in the deploy step due to the removal of requirements.txt.

[jaraco]

It should now be on PyPI.

[jakirkham]

Seems to not have been deployed as that change was not on a tag. Would it make sense to add a 36.0.1 tag?

ref: https://travis-ci.org/pypa/setuptools/jobs/237980870#L381

[nicoddemus]

IMHO it would make sense to just upload the release manually using twine, as it was a deployment problem and not a problem with the package.

[jaraco]

That's weird. I definitely ran the commands to cut a release. Oh, I ran setup.py release which built the dists but didn't upload them. :/

[jaraco]

Using twine, as recommended, the dists are now (verifiably) in PyPI.

[jakirkham]

Great thanks for doing this.

[alalazo]

@jaraco Hi! Another developer from Spack doing some necromancy to maybe resurrect an old issue 💀 We were hit by this problem today:

• Style checks are currently broken on develop spack/spack#31109

which was triggered by the new 64.0.0 release. The gist of it seems to be that, trying to install from source setuptools 62.3.2 with the following command:

/tmp/py38/bin/python3 '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/culpo/.spack/bootstrap/store/linux-ubuntu20.04-x86_64/gcc-9.4.0/py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne' '.'/tmp/py38/bin/python3 '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/culpo/.spack/bootstrap/store/linux-ubuntu20.04-x86_64/gcc-9.4.0/py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne' '.'

from a virtual environment that has setuptools 62.4.0 installed, fails when trying to create the metadata from the wheel. The error looks like the following:

Excerpt of the stacktrace

[ ... ]
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-z64i02nz
Created temporary directory: /tmp/pip-build-tracker-x6romquo
Initialized build tracking at /tmp/pip-build-tracker-x6romquo
Created build tracker: /tmp/pip-build-tracker-x6romquo
Entered build tracker: /tmp/pip-build-tracker-x6romquo
Created temporary directory: /tmp/pip-install-6b6sr5bk
Processing /tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
  Added file:///tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src to build tracker '/tmp/pip-build-tracker-x6romquo'
  Created temporary directory: /tmp/pip-modern-metadata-jcsudk0s
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info
  writing /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/top_level.txt
  writing /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 87, in <module>
      dist = setuptools.setup(**setup_params)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/dist.py", line 1229, in run_command
      super().run_command(command)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/dist_info.py", line 35, in run
      egg_info.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 308, in run
      self.find_sources()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 315, in find_sources
      mm.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 550, in run
      self.add_defaults()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 587, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/command/sdist.py", line 226, in add_defaults
      self._add_defaults_python()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/sdist.py", line 111, in _add_defaults_python
      build_py = self.get_finalized_command('build_py')
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
      cmd_obj.ensure_finalized()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
      self.finalize_options()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/build_py.py", line 33, in finalize_options
      orig.build_py.finalize_options(self)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/command/build_py.py", line 43, in finalize_options
      self.set_undefined_options('build',
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
      src_cmd_obj = self.distribution.get_command_obj(src_cmd)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 858, in get_command_obj
      klass = self.get_command_class(command)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/dist.py", line 966, in get_command_class
      self.cmdclass[command] = cmdclass = ep.load()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_vendor/importlib_metadata/__init__.py", line 194, in load
      module = import_module(match.group('module'))
    File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'setuptools.command.build'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /tmp/py38/bin/python3 /tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmppj2rk5qg
  cwd: /tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[ ... ]

and from:

ModuleNotFoundError: No module named 'setuptools.command.build'
  error: subprocess-exited-with-error

I got the impression that the "system" setuptools v62.4.0 was injected by pip during the install process. Part of it is maybe our fault, since we explicitly pass:

--no-build-isolation

for good reasons explained above, but since setuptools specifies no requirements:

[build-system]
requires = []
build-backend = "setuptools.build_meta"
backend-path = ["."]

I'm uncertain if this is a bug on setuptools, or on pip or or just our misunderstanding of how setuptools should be bootstrapped with PEP 518 now in place. Do you see better ways to bootstrap setuptools from sources?

[abravalheri]

Hi @alalazo, there might be some variation of commands between versions of setuptools (that is expected as setuptools development progresses).
This is the case of setuptools.command.build, which does not exist for older versions.

Ideally we want setuptools to build itself without the help of older/newer versions, this should be guaranteed by the frontend/installer according to PEP 517. There is not much we can do in setuptools other than configuring pyproject.toml as you pointed out.

I am not sure how pip handles the --no-build-isolation, but for normal builds some paths might leak into the build environment depending on the operating system (e.g. Debian-based).

---

Just as a curiosity (that might help us to understand what is going on), would you face the same error if you use build to create a wheel? (It also has the --no-isolation flag).

[abravalheri]

I run some experiments, and I think the problem the problem is the following:

• It is uncommon, but newer versions of setuptools may have a different set of entry-points. This is part of the natural evolution of the project (this will be the case for example when we implement PEP 660).

• If you run the build with pip/--no-build-isolation on pip or build/--no-isolation, existing entries of PYTHONPATH are left behind. This means that existing setuptools-*.dist-info directories will be picked up by importlib.metadata when using entry-points for auto-discovery.

• Entry-points from different versions can be incompatible and cause this type of problem.

I don't know if we can support this kind of usage right now (maybe we can do it in the future). For the time being I recommend bootstrapping setuptools in an environment without setuptools, or use build isolation.

[alalazo]

Just as a curiosity (that might help us to understand what is going on), would you face the same error if you use build to create a wheel? (It also has the --no-isolation flag).

I tried some manual experiments and build works fine both with isolation, and without if I take care of removing the system setuptools:

$ ls
changelog.d  conftest.py  _distutils_hack  exercises.py  LICENSE      PKG-INFO       pyproject.toml  README.rst  setup.py    setuptools.egg-info  tox.ini
CHANGES.rst  dist         docs             launcher.c    MANIFEST.in  pkg_resources  pytest.ini      setup.cfg   setuptools  tools

$ python -m build --no-isolation
[ ... ]
Successfully built setuptools-62.3.2.tar.gz and setuptools-62.3.2-py3-none-any.whl

This is not the case with pip¹, which makes me think pip might be at fault here?

Another data point is that currently we rely heavily on setting PYTHONPATH ourselves. Since we install each package into its own hashed directory (see #980 (comment) for some background of the Spack project) and we want control of how things are built, PYTHONPATH is the mechanism by which we ensure a build environment with --no-isolation finds its requirements.

Footnotes

1. I tried pip 20.0.2 and 22.1.2 ↩

[abravalheri]

I am actually impressed that the experiment with build works. I did not manage to make it work for me. I tried the following:

> docker run --rm -it fedora:36

dnf install -y git
python3 -m ensurepip
python3 -m pip install -U pip setuptools build
python3 -m pip list
# ...
# build      0.8.0
# pip        22.1.2
# setuptools 62.4.0
# ...

git clone --branch v62.3.2 https://github.com/pypa/setuptools /tmp/setuptools
cd /tmp/setuptools
python3 -m build --no-isolation
# ...
# ModuleNotFoundError: No module named 'setuptools.command.build'

So I was just thinking that it is a conceptual incompatibility...

[alalazo]

@abravalheri You're correct, build works fine only with isolation - or without if setuptools is not installed on the system.

[alalazo]

I hacked with my virtual environment and I confirm that the issue is indeed #980 (comment) Just before the failure these variables are set:

PEP517_BACKEND_PATH=/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
PEP517_BUILD_BACKEND=setuptools.build_meta

and they seem to respect what pyproject.toml prescribes. Inspecting the entry points though, one could see those stemming from the setuptools v62.4.0 installed on the system.