Python wheels impose vague and undocumented restrictions on the use of C++ features

[olupton]

Overview

While working on #1922 I started seeing linker errors:

nrnmain.cpp:(.text.startup+0x98): undefined reference to `nrnmpi_load[abi:cxx11](int)

in the CI jobs that test the Linux Python wheels (which are also built as part of the CI). The relevant change in that PR was that I changed the return type of that function from char* to std::string.

In general, replacing C-style strings and manual memory management with standard types such as std::string is a Good Thing ™️, so it's unfortunate that this causes problems with use cases that we consider important (pip install neuron on Linux...). If supporting these use cases forbids the use of certain C++ features or types in certain places, this should be made explicit and documented.

Detailed description

The problem is that the Python wheels are built using an image based on manylinux2014 that is based on CentOS7, and that this only supports using a pre-C++11 ABI for std::string (link, link, ...). This means that the old ABI is used for code in libnrniv.so, which is shipped inside the wheels.

The typical workflow of running nrnivmodl after pip install neuron on a "user" machine builds C++ sources using the compiler toolchain that is installed on that "user" machine, which must support at least C++17 and will default to using a C++11-compatible std::string implementation. This causes link errors if any code compiled on the user machine tries to refer to symbols inside libnrniv.so that use std::string (for example, a function that returns std::string).

Trying to force the use of the old ABI on user machines inside nrnivmodl would probably be fragile, and would not avoid issues with linking to other software on the user machine that uses the new ABI. Dropping support for manylinux2014 would(?) cause issues with pip install neuron using vanilla Python 3.7.x.

These issues can be avoided by only using a restricted subset of C++ in functions/variables that may be called from code compiled on the user machine, but there is no(?) well-defined list of these functions/variables or enforcement mechanism beyond hoping that the wheel-based CI pipelines catch any infractions.

Further information

Some other links that may be of interest:

• Building wheels : manylinux1 or manylinux2010? #511

[olupton]

This came up again in #1929, where an intermediate commit had a non-inline version of https://github.com/neuronsimulator/nrn/blob/4ac3ed9e46a29af193c807c50c101f88537a83e1/src/nrniv/backtrace_utils.h#L13-L24 built into libnrniv.so that caused linker errors downstream when using the Python wheels and nrnivmodl.

[olupton]

The latest manifestation of this in #2027 involves a class with an std::string member.

struct foo {
  std::string m_name;
  short m_number;
};

which cannot safely be passed between code compiled as part of the wheel (libnrniv.so) and inside nrnivmodl -- specifically attempting to access m_number leads to junk results.

[olupton]

Note that in BlueBrain/libsonata#273 we seem to have come across another, this time std::regex-related, incarnation of ABI issues when running a mix of manylinux2014 and modern compiled code in a single process.