diff --git a/CI.md b/CI.md index cd964a418..abcd6d889 100644 --- a/CI.md +++ b/CI.md @@ -1,11 +1,12 @@ This is a summary of the Python versions and platforms covered by the different CI platforms: -| | 3.5 | 3.6 | 3.7 | 3.8 | -|----------|------------------|------------------|---------------------------------------------------|------------------| -| Linux | Travis CI | CircleCI | AppVeyor / GitHub Actions | Azure Pipelines | -| macOS | Azure Pipelines | CircleCI | AppVeyor / Travis CI¹ / CircleCI / GitHub Actions | Azure Pipelines | -| Windows | TravisCI | Azure Pipelines | AppVeyor / GitHub Actions | Azure Pipelines | +| | 3.5 | 3.6 | 3.7 | 3.8 | +|----------|------------------|------------------|----------------------------------------------------|------------------| +| Linux | Travis CI | CircleCI | AppVeyor² / GitHub Actions | Azure Pipelines | +| macOS | Azure Pipelines | CircleCI | AppVeyor² / Travis CI¹ / CircleCI / GitHub Actions | Azure Pipelines | +| Windows | TravisCI | Azure Pipelines | AppVeyor² / GitHub Actions | Azure Pipelines | > ¹ Python version not really pinned, but dependent on the (default) version of image used. +> ² AppVeyor only runs the "basic" test to reduce load. Non-x86 architectures are covered on Travis CI using Python 3.5. diff --git a/docs/cpp_standards.md b/docs/cpp_standards.md index 9ac965845..5dd836142 100644 --- a/docs/cpp_standards.md +++ b/docs/cpp_standards.md @@ -30,6 +30,14 @@ For more details see https://en.cppreference.com/w/cpp/compiler_support, https:/ ## Windows and Python 2.7 -Visual C++ for Python 2.7 does not support modern standards of C++. When building on Appveyor, you will need to either use the "Visual Studio 2017" or "Visual Studio 2019" image, but Python 2.7 is not supported on these images - skip it by setting `CIBW_SKIP=cp27-win*`. +Visual C++ for Python 2.7 does not support modern C++ standards (i.e., C++11 and later). When building on Appveyor, you will need to either use the "Visual Studio 2017" or "Visual Studio 2019" image, but Python 2.7 is not supported on these images - skip it by setting `CIBW_SKIP=cp27-win*`. -There is an optional workaround for this, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install a newer "Microsoft Visual C++ Redistributable", since the newer C++ standard libraries are not included by default with the Python 2.7 installation. +There is an optional workaround for this, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install [a newer "Microsoft Visual C++ Redistributable"](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads), since the newer C++ standard library's binaries are not included by default with the Python 2.7 installation. + +Forcing `distutils` or `setuptools` to use a more recent version of MSVC that supports modern C++ can be done in the following way: + +1. Set the environment variables `DISTUTILS_USE_SDK=1` and `MSSdk=1`. These two environment variables will tell `distutils`/`setuptools` to not search and set up a build environment that uses Visual C++ for Python 2.7 (aka. MSVC 9). +2. Set up the build Visual Studio build environment you want to use, making sure that e.g. `PATH` contains `cl`, `link`, etc. + - Usually, this can be done through `vcvarsall.bat x86` or `vcvarsall.bat x64`. The exact location of this file depends on the installation, but the default path for VS 2019 Community (e.g. used in AppVeyor's `Visual Studio 2019` image) is `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat`. **Note**: `vcvarsall.bat` changes the environment variables, so this cannot be run in a subprocess/subshell and consequently running `vsvarsall.bat` in `CIBW_BEFORE_BUILD` does not have any effect. + - In Azure Pipelines, [a `VSBuild` task is available](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/visual-studio-build) and GitHub Actions has [an action `microsoft/setup-msbuild`](https://github.com/microsoft/setup-msbuild) that help setting up the Visual Studio environment. +3. Next, call `cibuildwheel`. Unfortunately, MSVC has separate toolchains for compiling 32-bit and 64-bit, so you will need to run `cibuildwheel` twice: once with `CIBW_BUILD=*-win32` after setting up the `x86` build environment, and once with `CIBW_BUILD=*-win_amd64` in a `x64` enviroment (see previous step). diff --git a/test/10_cpp_standards/cibuildwheel_test.py b/test/10_cpp_standards/cibuildwheel_test.py index 72a8f6765..3a1e57d57 100644 --- a/test/10_cpp_standards/cibuildwheel_test.py +++ b/test/10_cpp_standards/cibuildwheel_test.py @@ -7,7 +7,7 @@ project_dir = os.path.dirname(__file__) -def test_cpp11(tmp_path): +def test_cpp11(): # This test checks that the C++11 standard is supported # VC++ for Python 2.7 does not support modern standards @@ -25,14 +25,12 @@ def test_cpp14(): # VC++ for Python 2.7 does not support modern standards # The manylinux1 docker image does not have a compiler which supports C++11 - # Python 3.4 and 3.5 are compiled with MSVC 10, which does not support C++14 - add_env = {'CIBW_SKIP': 'cp27-win* pp27-win32 cp35-win*', 'CIBW_ENVIRONMENT': 'STANDARD=14'} + add_env = {'CIBW_SKIP': 'cp27-win* pp27-win32', 'CIBW_ENVIRONMENT': 'STANDARD=14'} actual_wheels = utils.cibuildwheel_run(project_dir, add_env=add_env) expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0') if 'cp27-cp27m-win' not in w - and 'pp27-pypy_73-win32' not in w - and 'cp35-cp35m-win' not in w] + and 'pp27-pypy_73-win32' not in w] assert set(actual_wheels) == set(expected_wheels) @@ -42,11 +40,10 @@ def test_cpp17(): # Python and PyPy 2.7 use the `register` keyword which is forbidden in the C++17 standard # The manylinux1 docker image does not have a compiler which supports C++11 - # Python 3.5 and PyPy 3.6 are compiled with MSVC 10, which does not support C++17 if os.environ.get('APPVEYOR_BUILD_WORKER_IMAGE', '') == 'Visual Studio 2015': pytest.skip('Visual Studio 2015 does not support C++17') - add_env = {'CIBW_SKIP': 'cp27-win* pp27-win32 cp35-win* pp36-win32', 'CIBW_ENVIRONMENT': 'STANDARD=17'} + add_env = {'CIBW_SKIP': 'cp27-win* pp27-win32', 'CIBW_ENVIRONMENT': 'STANDARD=17'} if utils.platform == 'macos': add_env['MACOSX_DEPLOYMENT_TARGET'] = '10.13' @@ -54,8 +51,56 @@ def test_cpp17(): actual_wheels = utils.cibuildwheel_run(project_dir, add_env=add_env) expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0', macosx_deployment_target='10.13') if 'cp27-cp27m-win' not in w - and 'pp27-pypy_73-win32' not in w - and 'cp35-cp35m-win' not in w - and 'pp36-pypy36_pp73-win32' not in w] + and 'pp27-pypy_73-win32' not in w] + + assert set(actual_wheels) == set(expected_wheels) + + +def test_cpp17_py27_modern_msvc_workaround(): + # This test checks the workaround for building Python 2.7 wheel with MSVC 14 + + if utils.platform != 'windows': + pytest.skip('the test is only relevant to the Windows build') + + if os.environ.get('APPVEYOR_BUILD_WORKER_IMAGE', '') == 'Visual Studio 2015': + pytest.skip('Visual Studio 2015 does not support C++17') + + # VC++ for Python 2.7 (i.e., MSVC 9) does not support modern standards + # This is a workaround which forces distutils/setupstools to a newer version + # Wheels compiled need a more modern C++ redistributable installed, which is not + # included with Python: see documentation for more info + # DISTUTILS_USE_SDK and MSSdk=1 tell distutils/setuptools that we are adding + # MSVC's compiler, tools, and libraries to PATH ourselves + add_env = {'CIBW_ENVIRONMENT': 'STANDARD=17', + 'DISTUTILS_USE_SDK': '1', 'MSSdk': '1'} + + # Use existing setuptools code to run Visual Studio's vcvarsall.bat and get the + # necessary environment variables, since running vcvarsall.bat in a subprocess + # does not keep the relevant environment variables + # There are different environment variables for 32-bit/64-bit targets, so we + # need to run cibuildwheel twice, once for 32-bit with `vcvarsall.bat x86, and + # once for 64-bit with `vcvarsall.bat x64` + # In a normal CI setup, just run vcvarsall.bat before running cibuildwheel and set + # DISTUTILS_USE_SDK and MSSdk + import setuptools + + def add_vcvars(prev_env, platform): + vcvarsall_env = setuptools.msvc.msvc14_get_vc_env(platform) + env = prev_env.copy() + for vcvar in ['path', 'include', 'lib']: + env[vcvar] = vcvarsall_env[vcvar] + return env + + add_env_x86 = add_vcvars(add_env, 'x86') + add_env_x86['CIBW_BUILD'] = '?p27-win32' + actual_wheels = utils.cibuildwheel_run(project_dir, add_env=add_env_x86) + + add_env_x64 = add_vcvars(add_env, 'x64') + add_env_x64['CIBW_BUILD'] = 'cp27-win_amd64' + actual_wheels += utils.cibuildwheel_run(project_dir, add_env=add_env_x64) + + expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0') + if 'cp27-cp27m-win' in w + or 'pp27-pypy_73-win32' in w] assert set(actual_wheels) == set(expected_wheels)