Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault with ipopt 3.14.12_1, casadi 3.6.3_0 and libgomp >= 13 on Linux #91

Closed
1 task done
traversaro opened this issue Sep 12, 2023 · 9 comments · Fixed by conda-forge/ctng-compilers-feedstock#117
Closed
1 task done

Segfault with ipopt 3.14.12_1, casadi 3.6.3_0 and libgomp >= 13 on Linux #91

traversaro opened this issue Sep 12, 2023 · 9 comments · Fixed by conda-forge/ctng-compilers-feedstock#117
Labels
bug

Comments

@traversaro
Copy link
Contributor

traversaro commented Sep 12, 2023 •
edited
Loading

TL;DR: A workaround is to pin libgomp to 12, i.e. mamba install libgomp=12.

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

traversaro@IITICUBLAP257:~/test$  mamba create -n casadi casadi
traversaro@IITICUBLAP257:~/test$   mamba activate casadi
(casadi) traversaro@IITICUBLAP257:~/test$  wget https://raw.githubusercontent.com/conda-forge/casadi-feedstock/main/recipe/ipopt_nl.py
(casadi) traversaro@IITICUBLAP257:~/test$ python ipopt_nl.py
Segmentation fault
(casadi) traversaro@IITICUBLAP257:~/test$ mamba install ipopt=3.14.12=*_0
(casadi) traversaro@IITICUBLAP257:~/test$ python ipopt_nl.py

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.12, running with linear solver MUMPS 5.2.1.

Installed packages

(casadi) traversaro@IITICUBLAP257:~/test$ mamba list
# packages in environment at /home/traversaro/miniforge3/envs/casadi:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
ampl-mp                   3.1.0             h2cc385e_1006    conda-forge
brotli-python             1.1.0           py311hb755f60_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
casadi                    3.6.3           py311hff1730f_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
ipopt                     3.14.12              he6d3896_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_0    conda-forge
libgfortran-ng            13.2.0               h69a702a_0    conda-forge
libgfortran5              13.2.0               ha4646dd_0    conda-forge
libgomp                   13.2.0               h807b86a_0    conda-forge
libhwloc                  2.9.1           nocuda_h7313eea_6    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           18_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libosqp                   0.6.3                h59595ed_0    conda-forge
libqdldl                  0.1.5                h27087fc_1    conda-forge
libspral                  2023.08.02           h2baf039_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.5               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
mumps-include             5.2.1               ha770c72_11    conda-forge
mumps-seq                 5.2.1               h2104b81_11    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.2          py311h64a7726_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
proxsuite                 0.3.7           py311h9547e67_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.5          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
scipy                     1.11.2          py311h64a7726_0    conda-forge
scotch                    6.0.9                hb2e6521_2    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
simde                     0.7.6                h00ab1b0_0    conda-forge
tinyxml2                  9.0.0                h9c3ff4c_2    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unixodbc                  2.3.12               h661eb56_0    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge

Environment info

(casadi) traversaro@IITICUBLAP257:~/test$ mamba info

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


     active environment : casadi
    active env location : /home/traversaro/miniforge3/envs/casadi
            shell level : 1
       user config file : /home/traversaro/.condarc
 populated config files : /home/traversaro/miniforge3/.condarc
                          /home/traversaro/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.2=0
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/traversaro/miniforge3  (writable)
      conda av data dir : /home/traversaro/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/traversaro/miniforge3/pkgs
                          /home/traversaro/.conda/pkgs
       envs directories : /home/traversaro/miniforge3/envs
                          /home/traversaro/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
@traversaro traversaro added the bug label Sep 12, 2023
@traversaro
Copy link
Contributor Author

Originally reported in ami-iit/hippopt#8 (comment) .

@traversaro
Copy link
Contributor Author

traversaro commented Sep 12, 2023 •
edited
Loading

I reproduced the failure in CI in #90 and #91 (with and without re-render)

2023-09-12T15:21:27.8690455Z + python ipopt_nl.py
2023-09-12T15:21:27.8691351Z /home/conda/feedstock_root/build_artifacts/casadi_1694531457704/test_tmp/run_test.sh: line 7:  2965 Segmentation fault      (core dumped) python ipopt_nl.py
2023-09-12T15:21:28.3353626Z Tests failed for casadi-3.6.3-py310h528c296_1.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

@traversaro
Copy link
Contributor Author

On C++, there is no segfault, the segfault is only on Python. I was able to obtain a really simple reproducer:

python -c "import ctypes; ctypes._dlopen('libcasadi_nlpsol_ipopt.so', 9)"

or even simpler:

python -c "import ctypes; ctypes._dlopen('libcasadi_nlpsol_ipopt.so', 8)"

The flags are:

RTLD_NOW: 2
RTLD_LAZY: 1
RTLD_GLOBAL: 256
RTLD_LOCAL: 0
RTLD_DEEPBIND: 8

So this seems some issue of Python with RTLD_DEEPBIND and ipopt linked with spral.

@traversaro
Copy link
Contributor Author

Exploring via ldd which library is segfaulting when loaded via RTLD_DEEPBIND, it turns out that it is libgomp, see:

 python -c "import ctypes; ctypes._dlopen('/home/traversaro/miniforge3/envs/hippoptdev/lib/./libgomp.so.1', 8)"

None of the libraries linked by libgomp have this problem, so the issue is probably due to libgomp itself. The issue was introduced in conda-forge/ipopt-feedstock#94 as since that PR ipopt links spral that links libgomp.

@traversaro
Copy link
Contributor Author

Indeed, the backtrace indicate a failure inside a libgomp function:

(gdb) bt
#0  initialize_env () at ../../../libgomp/env.c:2062
#1  0x00007ffff7fc947e in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fffffffc1f8, env=env@entry=0x7fffffffc218)
    at ./elf/dl-init.c:70
#2  0x00007ffff7fc9568 in call_init (env=0x7fffffffc218, argv=0x7fffffffc1f8, argc=3, l=<optimized out>) at ./elf/dl-init.c:33
#3  _dl_init (main_map=0x555555b8e620, argc=3, argv=0x7fffffffc1f8, env=0x7fffffffc218) at ./elf/dl-init.c:117
#4  0x00007ffff7e09c85 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>)
    at ./elf/dl-error-skeleton.c:182
#5  0x00007ffff7fd0ff6 in dl_open_worker (a=0x7fffffffb910) at ./elf/dl-open.c:808

@traversaro
Copy link
Contributor Author

From https://conda-forge.org/feedstock-outputs/ I get that libgomp is produced by https://github.com/conda-forge/ctng-compilers-feedstock , and as there is no specific patch for that file (see https://github.com/conda-forge/ctng-compilers-feedstock/tree/845b9c62143c4c2b2af8e35736e64aa29d8c3e05/recipe/patches ), so the failure is in https://github.com/gcc-mirror/gcc/blob/releases/gcc-13.2.0/libgomp/env.c#L2062 .

@traversaro
Copy link
Contributor Author

The issue happens with libgomp 13.2.0 and 13.1.0, but not with libgomp 12.* .

@traversaro traversaro mentioned this issue Sep 13, 2023
Closed
1 task
@traversaro
Copy link
Contributor Author

@SDafarra a simple workaround is to install libgomp (if you want to use spral enabled in the ipopt 3.14.12_1 build).

@S-Dafarra S-Dafarra mentioned this issue Sep 13, 2023
Closed
This was referenced Sep 19, 2023
Closed
Closed
@traversaro traversaro changed the title Segfault with ipopt 3.14.12_1 and casadi 3.6.3_0 on Linux Segfault with ipopt 3.14.12_1, casadi 3.6.3_0 and libgomp >= 13 on Linux Sep 20, 2023
@traversaro traversaro mentioned this issue Sep 20, 2023
Merged
5 tasks
@traversaro
Copy link
Contributor Author

The crash is fixed, but a problem remains related to the use of ipopt+spral with CasADi Python bindings: #93 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Backport fix for upstream issue 111413 to GCC13 traversaro/ctng-compilers-feedstock
1 participant
@traversaro