Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

setuptools 60+ breaks 3 distutils tests #1981

Closed
hroncok opened this issue Mar 15, 2022 · 4 comments · Fixed by #1984
Closed

setuptools 60+ breaks 3 distutils tests #1981

hroncok opened this issue Mar 15, 2022 · 4 comments · Fixed by #1984

Comments

@hroncok
Copy link
Contributor

hroncok commented Mar 15, 2022 •
edited
Loading

When we updated setuptools to 60+ in Fedora, it broke various packages. Setuptools now hijacks the distutils module and replaces it with its own. As such, it does some inconsistent things, such as pypa/setuptools#3143

Either way, I observe that 3 pythran tests consistently fail unless I set SETUPTOOLS_USE_DISTUTILS=stdlib (which disables this setuptools feature).

FAILED pythran/tests/test_distutils.py::TestDistutils::test_setup_build - sub...
FAILED pythran/tests/test_distutils.py::TestDistutils::test_setup_build2 - su...
FAILED pythran/tests/test_distutils.py::TestDistutils::test_setup_build3 - Fi...

________________________ TestDistutils.test_setup_build ________________________
[gw5] linux -- Python 3.10.2 /usr/bin/python3
self = <pythran.tests.test_distutils.TestDistutils testMethod=test_setup_build>
    def test_setup_build(self):
        check_call(['python', 'setup.py', 'build'],
                   cwd=os.path.join(cwd, 'test_distutils'))
        check_call(['python', 'setup.py', 'install', '--prefix=demo_install'],
                   cwd=os.path.join(cwd, 'test_distutils'))
    
        base = os.path.join(cwd, 'test_distutils', 'demo_install',)
        libdir = os.path.join(base, 'lib')
        if not os.path.isdir(libdir):
            libdir = os.path.join(base, 'lib64')
>       check_call(['python', '-c', 'import demo'],
                   cwd=os.path.join(libdir, python_version, 'site-packages'))
pythran/tests/test_distutils.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
popenargs = (['python', '-c', 'import demo'],)
kwargs = {'cwd': '/builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils/demo_install/lib64/python3.10/site-packages'}
retcode = 1, cmd = ['python', '-c', 'import demo']
    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['python', '-c', 'import demo']' returned non-zero exit status 1.
/usr/lib64/python3.10/subprocess.py:369: CalledProcessError
----------------------------- Captured stdout call -----------------------------
running build
running build_ext
building 'demo' extension
INFO: C compiler: g++ -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC
creating build
creating build/temp.linux-x86_64-3.10
INFO: compile options: '-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_OPENBLAS -I/usr/include/flexiblas -I/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib/python3.10/site-packages/pythran -I/usr/lib64/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -c'
extra options: '-std=c++11 -fno-math-errno -fvisibility=hidden -fno-wrapv -Wno-unused-function -Wno-int-in-bool-context -Wno-unknown-warning-option -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
INFO: g++: a.cpp
creating build/lib.linux-x86_64-3.10
INFO: g++ -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/pythran-0.11.0/.package_note-pythran-0.11.0-3.fc37.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-3.10/a.o -L/usr/lib64 -L/usr/lib64 -lflexiblas -lflexiblas -lflexiblas -o build/lib.linux-x86_64-3.10/demo.cpython-310-x86_64-linux-gnu.so -fvisibility=hidden -Wl,-strip-all -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/pythran-0.11.0/.package_note-pythran-0.11.0-3.fc37.x86_64.ld
running install
Checking .pth file support in demo_install/lib64/python3.10/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: demo_install/lib64/python3.10/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
    demo_install/lib64/python3.10/site-packages/
and your PYTHONPATH environment variable currently contains:
    '/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib/python3.10/site-packages'
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:
  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
running bdist_egg
running egg_info
creating demo.egg-info
writing demo.egg-info/PKG-INFO
writing dependency_links to demo.egg-info/dependency_links.txt
writing top-level names to demo.egg-info/top_level.txt
writing manifest file 'demo.egg-info/SOURCES.txt'
reading manifest file 'demo.egg-info/SOURCES.txt'
writing manifest file 'demo.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.10/demo.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
warning: install_lib: byte-compiling is disabled, skipping.
creating stub loader for demo.cpython-310-x86_64-linux-gnu.so
warning: install_lib: byte-compiling is disabled, skipping.
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying demo.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/demo-1.0-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing demo-1.0-py3.10-linux-x86_64.egg
Copying demo-1.0-py3.10-linux-x86_64.egg to /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils/demo_install/lib64/python3.10/site-packages
Installed /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils/demo_install/lib64/python3.10/site-packages/demo-1.0-py3.10-linux-x86_64.egg
Processing dependencies for demo==1.0
Finished processing dependencies for demo==1.0
----------------------------- Captured stderr call -----------------------------
/usr/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py:160: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'demo'
_______________________ TestDistutils.test_setup_build2 ________________________
[gw0] linux -- Python 3.10.2 /usr/bin/python3
self = <pythran.tests.test_distutils.TestDistutils testMethod=test_setup_build2>
    def test_setup_build2(self):
        check_call(['python', 'setup.py', 'build'],
                   cwd=os.path.join(cwd, 'test_distutils_packaged'))
        check_call(['python', 'setup.py', 'install', '--prefix=demo_install2'],
                   cwd=os.path.join(cwd, 'test_distutils_packaged'))
    
        base = os.path.join(cwd, 'test_distutils_packaged', 'demo_install2',)
        libdir = os.path.join(base, 'lib')
        if not os.path.isdir(libdir):
            libdir = os.path.join(base, 'lib64')
>       check_call(['python', '-c', 'import demo2.a'],
                   cwd=os.path.join(libdir, python_version, 'site-packages'))
pythran/tests/test_distutils.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
popenargs = (['python', '-c', 'import demo2.a'],)
kwargs = {'cwd': '/builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_packaged/demo_install2/lib64/python3.10/site-packages'}
retcode = 1, cmd = ['python', '-c', 'import demo2.a']
    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['python', '-c', 'import demo2.a']' returned non-zero exit status 1.
/usr/lib64/python3.10/subprocess.py:369: CalledProcessError
----------------------------- Captured stdout call -----------------------------
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/demo2
copying demo2/__init__.py -> build/lib.linux-x86_64-3.10/demo2
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
building 'demo2.a' extension
INFO: C compiler: g++ -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC
creating build/temp.linux-x86_64-3.10
INFO: compile options: '-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_OPENBLAS -I/usr/include/flexiblas -I/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib/python3.10/site-packages/pythran -I/usr/lib64/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -c'
extra options: '-std=c++11 -fno-math-errno -fvisibility=hidden -fno-wrapv -Wno-unused-function -Wno-int-in-bool-context -Wno-unknown-warning-option -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
INFO: g++: a.cpp
INFO: g++ -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/pythran-0.11.0/.package_note-pythran-0.11.0-3.fc37.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-3.10/a.o -L/usr/lib64 -L/usr/lib64 -lflexiblas -lflexiblas -lflexiblas -o build/lib.linux-x86_64-3.10/demo2/a.cpython-310-x86_64-linux-gnu.so -fvisibility=hidden -Wl,-strip-all -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/pythran-0.11.0/.package_note-pythran-0.11.0-3.fc37.x86_64.ld
running install
Checking .pth file support in demo_install2/lib64/python3.10/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: demo_install2/lib64/python3.10/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
    demo_install2/lib64/python3.10/site-packages/
and your PYTHONPATH environment variable currently contains:
    '/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib/python3.10/site-packages'
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:
  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
running bdist_egg
running egg_info
creating demo2.egg-info
writing demo2.egg-info/PKG-INFO
writing dependency_links to demo2.egg-info/dependency_links.txt
writing top-level names to demo2.egg-info/top_level.txt
writing manifest file 'demo2.egg-info/SOURCES.txt'
reading manifest file 'demo2.egg-info/SOURCES.txt'
writing manifest file 'demo2.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/demo2
copying build/lib.linux-x86_64-3.10/demo2/__init__.py -> build/bdist.linux-x86_64/egg/demo2
copying build/lib.linux-x86_64-3.10/demo2/a.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/demo2
warning: install_lib: byte-compiling is disabled, skipping.
creating stub loader for demo2/a.cpython-310-x86_64-linux-gnu.so
warning: install_lib: byte-compiling is disabled, skipping.
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying demo2.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo2.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo2.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo2.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/demo2-1.0-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing demo2-1.0-py3.10-linux-x86_64.egg
Copying demo2-1.0-py3.10-linux-x86_64.egg to /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_packaged/demo_install2/lib64/python3.10/site-packages
Installed /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_packaged/demo_install2/lib64/python3.10/site-packages/demo2-1.0-py3.10-linux-x86_64.egg
Processing dependencies for demo2==1.0
Finished processing dependencies for demo2==1.0
----------------------------- Captured stderr call -----------------------------
/usr/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py:160: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'demo2'
_______________________ TestDistutils.test_setup_build3 ________________________
[gw1] linux -- Python 3.10.2 /usr/bin/python3
self = <pythran.tests.test_distutils.TestDistutils testMethod=test_setup_build3>
    def test_setup_build3(self):
        check_call(['python', 'setup.py', 'build'],
                   cwd=os.path.join(cwd, 'test_distutils_numpy'))
        check_call(['python', 'setup.py', 'install', '--prefix=demo_install3'],
                   cwd=os.path.join(cwd, 'test_distutils_numpy'))
    
        base = os.path.join(cwd, 'test_distutils_numpy', 'demo_install3',)
        libdir = os.path.join(base, 'lib')
        if not os.path.isdir(libdir):
            libdir = os.path.join(base, 'lib64')
>       check_call(['python', '-c', 'import a'],
                   cwd=os.path.join(libdir, python_version, 'site-packages',
                                    'demo3'))
pythran/tests/test_distutils.py:118: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.10/subprocess.py:364: in check_call
    retcode = call(*popenargs, **kwargs)
/usr/lib64/python3.10/subprocess.py:345: in call
    with Popen(*popenargs, **kwargs) as p:
/usr/lib64/python3.10/subprocess.py:966: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Popen: returncode: 255 args: ['python', '-c', 'import a']>
args = ['python', '-c', 'import a'], executable = b'python', preexec_fn = None
close_fds = True, pass_fds = ()
cwd = '/builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/demo_install3/lib64/python3.10/site-packages/demo3'
env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1, errwrite = -1
restore_signals = True, gid = None, gids = None, uid = None, umask = -1
start_new_session = False
    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/demo_install3/lib64/python3.10/site-packages/demo3'
/usr/lib64/python3.10/subprocess.py:1842: FileNotFoundError
----------------------------- Captured stdout call -----------------------------
running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building extension "demo3.a" sources
INFO: build_src: building npy-pkg config files
running build_ext
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using PythranBuildExt
INFO: CCompilerOpt.__init__[773] : load cache from file -> /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_ext.py
INFO: CCompilerOpt.__init__[784] : hit the file cache
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using PythranBuildExt
INFO: 
########### EXT COMPILER OPTIMIZATION ###########
INFO: Platform      : 
  Architecture: x64
  Compiler    : gcc
CPU baseline  : 
  Requested   : 'min'
  Enabled     : SSE SSE2 SSE3
  Flags       : -msse -msse2 -msse3
  Extra checks: none
CPU dispatch  : 
  Requested   : 'max -xop -fma4'
  Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
  Generated   : none
INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_ext.py
running install
Checking .pth file support in demo_install3/lib64/python3.10/site-packages
/usr/bin/python -E -c pass
TEST FAILED: demo_install3/lib64/python3.10/site-packages does NOT support .pth files
bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
    demo_install3/lib64/python3.10/site-packages
and your PYTHONPATH environment variable currently contains:
    '/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/pythran-0.11.0-3.fc37.x86_64/usr/lib/python3.10/site-packages'
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:
  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
running bdist_egg
running egg_info
running build_src
INFO: build_src
INFO: building extension "demo3.a" sources
INFO: build_src: building npy-pkg config files
writing demo3.egg-info/PKG-INFO
writing dependency_links to demo3.egg-info/dependency_links.txt
writing top-level names to demo3.egg-info/top_level.txt
reading manifest file 'demo3.egg-info/SOURCES.txt'
writing manifest file 'demo3.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using PythranBuildExt
INFO: CCompilerOpt.__init__[773] : load cache from file -> /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_ext.py
INFO: CCompilerOpt.__init__[784] : hit the file cache
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using PythranBuildExt
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/demo3
copying build/lib.linux-x86_64-3.10/demo3/a.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/demo3
warning: install_lib: byte-compiling is disabled, skipping.
creating stub loader for demo3/a.cpython-310-x86_64-linux-gnu.so
warning: install_lib: byte-compiling is disabled, skipping.
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying demo3.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo3.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo3.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying demo3.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/demo3-1.0-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing demo3-1.0-py3.10-linux-x86_64.egg
Copying demo3-1.0-py3.10-linux-x86_64.egg to /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/demo_install3/lib64/python3.10/site-packages
Installed /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/demo_install3/lib64/python3.10/site-packages/demo3-1.0-py3.10-linux-x86_64.egg
Processing dependencies for demo3==1.0
Finished processing dependencies for demo3==1.0
INFO: 
########### EXT COMPILER OPTIMIZATION ###########
INFO: Platform      : 
  Architecture: x64
  Compiler    : gcc
CPU baseline  : 
  Requested   : 'min'
  Enabled     : SSE SSE2 SSE3
  Flags       : -msse -msse2 -msse3
  Extra checks: none
CPU dispatch  : 
  Requested   : 'max -xop -fma4'
  Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
  Generated   : none
INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /builddir/build/BUILD/pythran-0.11.0/pythran/tests/test_distutils_numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_ext.py
----------------------------- Captured stderr call -----------------------------
/usr/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py:160: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
@hroncok
Copy link
Contributor Author

hroncok commented Mar 15, 2022

Looking through the logs, I am more confident this is actually caused by pypa/setuptools#3143 -- the fact that the testing packages are installed as eggs break some assumptions of the tests.

@mattip
Copy link
Contributor

mattip commented Apr 3, 2022

I am also seeing failing tests on a weekly cron job to test pythran HEAD + PyPy HEAD. Last night's run started failing the pythran/tests/test_distutils.py tests. It seems to be the same problem, but was working fine until this week. I wonder what changed.

serge-sans-paille added a commit that referenced this issue Apr 3, 2022
@serge-sans-paille
Harden distutils tests
e05750f 
Use correct executable when running import tests.
Should fix #1981
@serge-sans-paille serge-sans-paille mentioned this issue Apr 3, 2022
Merged
@serge-sans-paille
Copy link
Owner

@mattip / @hroncok I wonder if #1984 fixes this issue?

@hroncok
Copy link
Contributor Author

hroncok commented Apr 3, 2022

I haven't tested it, but I don't think that change makes any difference.

serge-sans-paille added a commit that referenced this issue Apr 3, 2022
@serge-sans-paille
Harden distutils tests
7ef26b9 
Use correct executable when running import tests.
Use pip install instead of setup.py install to test install.
Should fix #1981
serge-sans-paille added a commit that referenced this issue Apr 4, 2022
@serge-sans-paille
Harden distutils tests
934e31d 
Use correct executable when running import tests.
Use pip install instead of setup.py install to test install.
Should fix #1981
serge-sans-paille added a commit that referenced this issue Apr 4, 2022
@serge-sans-paille
Harden distutils tests
26a23b9 
Use correct executable when running import tests.
Use pip install instead of setup.py install to test install.
Should fix #1981
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Harden distutils tests serge-sans-paille/pythran
3 participants
@mattip @serge-sans-paille @hroncok