"Building Linux wheels for Python 3.8 requires a compiler (e.g gcc)."

[samuela]

The pip install process for slycot fails on most clean linux installs. Here are the results running in a circleci/python Docker container:

$ pip install slycot
Defaulting to user installation because normal site-packages is not writeable
Collecting slycot
  Using cached slycot-0.3.5.0.tar.gz (1.5 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy
  Using cached numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
Building wheels for collected packages: slycot
  Building wheel for slycot (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmprwu6q9xf
       cwd: /tmp/pip-install-djjch8io/slycot
  Complete output (84 lines):
  Not searching for unused variables given on the command line.
  CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-djjch8io/slycot/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
  Not searching for unused variables given on the command line.
  -- The C compiler identification is GNU 8.3.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc - works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is unknown
  CMake Error at CMakeLists.txt:4 (ENABLE_LANGUAGE):
    No CMAKE_Fortran_COMPILER could be found.

    Tell CMake where to find the compiler by setting either the environment
    variable "FC" or the CMake cache entry CMAKE_Fortran_COMPILER to the full
    path to the compiler, or to the compiler name if it is in the PATH.


  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-djjch8io/slycot/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
  See also "/tmp/pip-install-djjch8io/slycot/_cmake_test_compile/build/CMakeFiles/CMakeError.log".


  --------------------------------------------------------------------------------
  -- Trying "Ninja" generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying "Ninja" generator - failure
  --------------------------------------------------------------------------------



  --------------------------------------------------------------------------------
  -- Trying "Unix Makefiles" generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying "Unix Makefiles" generator - failure
  --------------------------------------------------------------------------------

  ********************************************************************************
  scikit-build could not get a working generator for your system. Aborting build.

  Building Linux wheels for Python 3.8 requires a compiler (e.g gcc).
  It can be installed using debian package manager:

    sudo apt-get install build-essential

  To build compliant wheels, consider using the manylinux system described in PEP-513.
  Get it with "dockcross/manylinux-x64" docker image:

    https://github.com/dockcross/dockcross#readme

  For more details, please refer to scikit-build documentation:

    http://scikit-build.readthedocs.io/en/latest/generators.html#linux

  ********************************************************************************
  ----------------------------------------
  ERROR: Failed building wheel for slycot
Failed to build slycot
ERROR: Could not build wheels for slycot which use PEP 517 and cannot be installed directly

But the thing is... running sudo apt-get install build-essential doesn't help at all. Installing build-essential and then re-running results in exactly the same error.

I don't have issues pip installing any other packages in the same container.

[samuela]

It's also worth mentioning that gcc is in fact installed in the container:

$ gcc
gcc: fatal error: no input files
compilation terminated.

[bnavigator]

Hi,

To quote from the version 0.3.5 README:

If a pre-complied binary (i.e. "wheel") is available it will be installed with no need for compilation. If not, pip will attempt to compile the package from source and thus the compilation dependencies will be required (scikit-build, gfortran, BLAS, etc.).

From your log:

  -- The C compiler identification is GNU 8.3.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc - works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is unknown
  CMake Error at CMakeLists.txt:4 (ENABLE_LANGUAGE):
    No CMAKE_Fortran_COMPILER could be found.

gcc 8.3.0 is found, but gfortran is not installed. The generic scikit-build message does not differentiate between the C and Fortran compiler.

[samuela]

Yeah, as I discovered the magic incantation is

sudo apt install gfortran libblas-dev liblapack-dev

[bnavigator]

... or any other more performant blas/lapack implementation

[orzechow]

I'm running into the same problem in a CI build job using Ubuntu 18.04 with ROS in a docker container, which installs slycot automatically via pip (as a dependency of our application code).

As the slycot pip build depends on gfortran, but doesn't provide it as dependency itself, the build fails.
Even, if I manually specify gfortran as additional dependency of my application, the build does not always succeed, because we can't define the order in which the dependencies are installed (rosdep only follows the dependency tree).

I'm not an expert on pip, but would it be possible to either

• provide pip wheels (at least for Ubuntu LTS versions) to prevent building from source or
• define gfortran as a dependency of slycot, install it via build scripts, … (as said, I'm not really aware of how pip works with this regards)?

[bnavigator]

I'm not an expert on pip, but would it be possible to either

• provide pip wheels (at least for Ubuntu LTS versions) to prevent building from source or

Wheels are not distro specific. When we provide wheels, which are linked with the blas/lapack libraries of one Ubuntu LTS, people will try to install them on different Linux distros and have a horrible user experience. The NumPy and SciPy wheels provide their own lapack/libraries and we would have to sync with them in order to avoid trouble.

• define gfortran as a dependency of slycot, install it via build scripts, … (as said, I'm not really aware of how pip works with this regards)?

As far as I am aware, pip only follows the Python package dependencies specified via setuptools. A Fortran compiler is not a Python package. It has to be provided through the operating system before running pip install. For Ubuntu, that is apt-get install gfortran as mentioned above.

I am not familiar with ROS containers. Here are two ideas, which you could try:

• Find a place or hook in your CI/container-creation/ROS-definition which can call apt-get install gfortran ... before the pip install your-application
• Create your own custom wheel in a separate Ubuntu environment with pip wheel and inject the file into the docker system

If you need pre-compiled packages, consider using the conda ecosystem instead of pip. Or use a distribution which provides distro specific packages. I know of Archlinux (AUR) and openSUSE Tumleweed.

[bnavigator]

Even, if I manually specify gfortran as additional dependency of my application, the build does not always succeed, because we can't define the order in which the dependencies are installed (rosdep only follows the dependency tree).

Hmm, that sounds more like you have specified a runtime dependency where a buildtime dependency should have been used. Somehow you must specify to rosdep that gfortran is a build requirement for Slycot.

[orzechow]

Hmm, that sounds more like you have specified a runtime dependency where a buildtime dependency should have been used. Somehow you must specify to rosdep that gfortran is a build requirement for Slycot.

Actually, I did specify gfortran as a build dependency of my ROS package:

<package format="3">
  <name>my_ros_pkg</name>
  […]
  <buildtool_depend>gfortran</buildtool_depend>
  <exec_depend>python-control-pip</exec_depend>
  <exec_depend>python-slycot-pip</exec_depend>
</package>

but rosdep is interpreting buildtool_depend as

build system tools which this package needs to build itself

so it thinks gfortran is needed to build my_ros_pkg, not to build python-slycot-pip.
Which makes sense, because gfortran is a build dependency of slycot – not my_ros_pkg.

That's why I'm asking why pip is not installing gfortran, if it needs it or how to get it to do so?

As a workaround I could try to setup a wrapper package with gfortran as build dependency and then add this wrapper package to my_ros_pkg.
Still, this feels hacky…

[bnavigator]

That's why I'm asking why pip is not installing gfortran, if it needs it or how to get it to do so?

Because pip can only install python packages. It is not meant to install system packages at all. If you run pip as root it gives you a stern warning.

I guess you have to fix the ROS package python-slycot-pip.

[orzechow]

You're right, pip wouldn't be able to install system packages anyways (And using pip as root is a bad idea, because it can break the system Python installation).

Unfortunately there is no ROS package python-slycot-pip I could fix. It is simply a dependency definition telling rosdep (the ROS dependency manager) to resolve it to the python-slycot pip package – probably assuming that all pip packages will be installed from wheels, if they contain binaries.
And it seems that I can't add build dependencies to this rosdep entry…

Btw. changing the OS is not an option, as ROS is (more or less) tightly coupled to Ubuntu.
Neither is conda supported by rosdep 😖

I'll take care to install gfortran in all our docker containers and physical machines then (won't hurt anyways) 💁‍♂️

Thanks for your explanations and efforts @bnavigator !

[bnavigator]

Unfortunately there is no ROS package python-slycot-pip I could fix.

You could create a python-slycot ROS package (or even a Debian/Ubuntu one).

I'll take care to install gfortran in all our docker containers and physical machines then (won't hurt anyways)

Don't forget cmake and the lapack/blas headers. Have a look at our GitHub Actions workflow for hints:

Slycot/.github/workflows/slycot-build-and-test.yml

Lines 88 to 99 in a40505a

	- name: Setup Ubuntu
	if: matrix.os == 'ubuntu'
	run: |
	sudo apt-get -y update
	sudo apt-get -y install gfortran cmake --fix-missing
	case ${{ matrix.bla_vendor }} in
	unset | Generic ) sudo apt-get -y install libblas-dev liblapack-dev ;;
	OpenBLAS ) sudo apt-get -y install libopenblas-dev ;;
	*)
	echo "bla_vendor option ${{ matrix.bla_vendor }} not supported"
	exit 1 ;;
	esac

Slycot/.github/workflows/slycot-build-and-test.yml

Lines 115 to 122 in a40505a

	- name: Build wheel
	env:
	BLA_VENDOR: ${{ matrix.bla_vendor }}
	CMAKE_GENERATOR: Unix Makefiles
	run: |
	if [[ $BLA_VENDOR = unset ]]; then unset BLA_VENDOR; fi
	python -m pip install --upgrade pip
	pip wheel -v -w . .

[orzechow]

Thanks for the hint!