[BUG] setuptools>=60 fails to find Python.h on Pyston

[BtbN]

setuptools version

setuptools>=60

Python version

Pyston 2.3.1

OS

Ubuntu 20.04, 21.10. Gentoo. Debian 11.

Additional environment information

Pyston-2.3.1 installed via pyenv inside of an isolated Docker-Container with no system Python-Components installed.

Description

Trying to install numpy results in an error about it not finding Python.h when calling distutils try_compile.

It does not give any indication on where it's looking for it, but another package, Pandas also exhibits this issue, though when calling gcc. It's passing -I/usr/local/pyenv/versions/pyston-2.3.1/include/python3.8 to gcc. While the correct path would be /usr/local/pyenv/versions/pyston-2.3.1/include/python3.8-pyston2.3.

Dumping sysconfig.get_paths() does show the correct include path, so that's not where it is getting it from.

This is a relatively recent issue. This worked a week ago, and downgrading to setuptools<60 makes the issue go away.

Expected behavior

Well, I'd expect a successful build.

How to Reproduce

• Setup a pyenv environment with latest Pyston (currentls 2.3.1).
• Upgrade pip/setuptools/wheel inside of that environment to latest versions.
• pip install numpy

Output

# pip install --no-cache-dir numpy
Collecting numpy
  Downloading numpy-1.21.5.zip (10.7 MB)
     |████████████████████████████████| 10.7 MB 4.1 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: numpy
  Building wheel for numpy (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/pyenv/versions/pyston-2.3.1/bin/python /usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp5il3akyy
       cwd: /tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9
  Complete output (95 lines):
  Processing numpy/random/_bounded_integers.pxd.in
  Processing numpy/random/mtrand.pyx
  Processing numpy/random/_sfc64.pyx
  Processing numpy/random/_mt19937.pyx
  Processing numpy/random/_philox.pyx
  Processing numpy/random/_pcg64.pyx
  Processing numpy/random/_common.pyx
  Processing numpy/random/_generator.pyx
  Processing numpy/random/_bounded_integers.pyx.in
  Processing numpy/random/bit_generator.pyx
  Cythonizing sources
  creating /tmp/tmpficqc93o/tmp
  creating /tmp/tmpficqc93o/tmp/tmpficqc93o
  non-existing path in 'numpy/distutils': 'site.cfg'
  running bdist_wheel
  running build
  running config_cc
  running config_fc
  running build_src
  creating build
  creating build/src.linux-x86_64-3.8
  creating build/src.linux-x86_64-3.8/numpy
  creating build/src.linux-x86_64-3.8/numpy/distutils
  creating build/src.linux-x86_64-3.8/numpy/core
  creating build/src.linux-x86_64-3.8/numpy/core/src
  creating build/src.linux-x86_64-3.8/numpy/core/src/npymath
  creating build/src.linux-x86_64-3.8/numpy/core/src/multiarray
  Running from numpy source directory.
  /tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/system_info.py:2026: UserWarning:
      Optimized (vendor) Blas libraries are not found.
      Falls back to netlib Blas library which has worse performance.
      A better performance should be easily gained by switching
      Blas library.
    if self._calc_info(blas):
  Warning: attempted relative import with no known parent package
  /tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
    warnings.warn(msg)
  Traceback (most recent call last):
    File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 230, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 267, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 448, in <module>
      setup_package()
    File "setup.py", line 440, in setup_package
      setup(**metadata)
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/core.py", line 169, in setup
      return old_setup(**new_attr)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build.py", line 61, in run
      old_build.run(self)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 144, in run
      self.build_sources()
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 161, in build_sources
      self.build_extension_sources(ext)
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 318, in build_extension_sources
      sources = self.generate_sources(sources, ext)
    File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 378, in generate_sources
      source = func(extension, build_dir)
    File "numpy/core/setup.py", line 434, in generate_config_h
      moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
    File "numpy/core/setup.py", line 44, in check_types
      out = check_types(*a, **kw)
    File "numpy/core/setup.py", line 289, in check_types
      raise SystemError(
  SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
  ----------------------------------------
  ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

For background, Setuptools intentionally made a breaking change in version 60, making the local copy of distutils the preferred version. As a result, any behaviors that rely on a patched or monkey-patched version of distutils in the standard library will not work. The temporary escape hatch is to set SETUPTOOLS_USE_DISTUTILS=stdlib.

There are known incompatibilities between Setuptools and numpy (see #2372).

Dumping sysconfig.get_paths() does show the correct include path

Unfortunately, the "include" paths presented by sysconfig are insufficient for distutils, which instead requires a path called "headers". Distutils works around this limitation by supplying the "headers" path for schemes that don't include it (see pypa/distutils#88 (comment)).

As I see it, this issue is essentially a duplicate of #2372 and numpy users should rely on SETUPTOOLS_USE_DISTUTILS=stdlib until numpy no longer relies on monkeypatching.