Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

[safijari]

Update for the weary traveler: You may be using a too old version of patchelf, try compiling it from source and using that instead.

I'm getting this error when I try to import the module after install my repaired wheel in a fresh VM (or trying to import it directly from the .so on the machine where it was built).

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

The wheel is build for this package (https://github.com/safijari/pyOpenKarto/blob/python-devel/setup.py) in python2.7. I run auditwheel and get:

(auditwheel_pipenv-zl3561H8) > $ auditwheel repair py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl --plat linux_x86_64
Repairing py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 -> .libsopenkarto/libstdc++-637c843a.so.6.0.25
Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libm-2.27.so -> .libsopenkarto/libm-2-24310d2a.27.so
Grafting: /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1 -> .libsopenkarto/libboost_thread-4e4bd896.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libc-2.27.so -> .libsopenkarto/libc-2-cd7c1a03.27.so
warning: working around a Linux kernel bug by creating a hole of 2101248 bytes in ‘.libsopenkarto/libc-2-cd7c1a03.27.so’
Grafting: /lib/x86_64-linux-gnu/libpthread-2.27.so -> .libsopenkarto/libpthread-2-1032040b.27.so
warning: working around a Linux kernel bug by creating a hole of 2076672 bytes in ‘.libsopenkarto/libpthread-2-1032040b.27.so’
Grafting: /lib/x86_64-linux-gnu/libgcc_s.so.1 -> .libsopenkarto/libgcc_s-b10d5179.so.1
Grafting: /lib/x86_64-linux-gnu/librt-2.27.so -> .libsopenkarto/librt-2-110b6497.27.so
Setting RPATH: openkarto.so to "$ORIGIN/.libsopenkarto"
Previous filename tags: linux_x86_64
No filename tags change needed.
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
No WHEEL info change needed.

Fixed-up wheel written to /home/jari/Pipenvs/karto/pyOpenKarto/dist/wheelhouse/py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl

Nothing seems wrong with that output as far as I can tell. Pls halp :(

Update: Hmm, I moved all the .so files elsewhere and moved them back one by one, once I moved back this file...

Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1

that's when I get the crash... hmm. Unsure if this is to blame as I have another package like this which doesn't use boost and gives the same crash after auditwheel.

[safijari]

Eeeeeeeeekh...

I think this (NixOS/patchelf#85) was needed, but the last release from patchelf (which is in the ubuntu repos from what I can tell) was made prior to that PR. Building patchelf from source fixed my woes.

[njsmith]

Where are you running auditwheel? Are you using the manylinux1 container?

[safijari]

I am not. I was having a lot of trouble getting all of my dependencies in Centos 5 and I since I have a very narrow list of target platforms (this is for private packages) I'm currently just running it in Ubuntu 16.04/18.04.

Where I stand right now: I can patch my wheels just fine in either 16.04 or 18.04 and the resulting wheel works fine in other installations of 16.04 or 18.04 respectively. From the readme I had assumed that something compiled on an older OS should work ok in a later OS but that's not my experience so far. 18.04 wheels on 16.04 give

ImportError: /usr/local/lib/python2.7/dist-packages/.libsopenkarto/libc-2-cd7c1a03.27.so: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

and 16.04 wheels in 18.04 straight up crash:

[1]    10563 illegal hardware instruction (core dumped)  ipython

This isn't the end of the world but any advice on this would be helpful. Should I try finding the oldest possible linux where I can actually build my extensions?

[njsmith]

OK, so yeah, it sounds like your original issue is that you were using an old and broken version of patchelf, which is unfortunate, but if you were sourcing patchelf from someone else then there's not a lot we can do...

The error for 18.04 wheels on 16.04 is expected – that's the linker detecting that you're using a library that assumes it's running on a newer system than 16.04 provides.

I don't offhand know why you're getting a SIGILL for the 16.04 wheels running on 18.04. Probably the next step would be to use gdb to debug where it's happening, and if you figure it out then it'd be nice to know in case someone else runs into it, but it's not a configuration we have a lot of experience with.

[safijari]

Okay, I'll try gdb and post back (actually is there a better place to have this discussion perchance?, feels out of place in this issue).

[safijari]

GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/jari/.local/share/virtualenvs/Pipenvs-IzK67E3O/bin/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from apriltags_eth import *

Program received signal SIGILL, Illegal instruction.
0x00007ffff7df4b9c in ?? () from /lib64/ld-linux-x86-64.so.2

I suspect this isn't sufficient information. I'm unfortunately not very skilled in gdb.

Note that the system on which this wheel was built and patched had

Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2

If that matters.

[safijari]

Out of curiosity I tried building on a 14.04 VM and bringing that over to my 18.04 machine, got a bus error this time. The same wheel works great in my 16.04 VM. I'm starting to wonder if this is a VM / non VM thing.

Edit 1: A colleague running 16.04 on the metal can run the VM generated packages so it's not something specifically related to processor types as I was thinking it might be. Back to drawing board.

Edit 2: A VM running 18.04 had the same error. I suppose I should give building using the manylinux1 docker image another college try.

[safijari]

Final update: sucked it up and built everything inside a manylinux1 container. Everything works great now.

[njsmith]

Glad to hear it all worked out in the end :-)

[jon-sch]

Should _verify_patchelf() be updated based on NixOS/patchelf#85? This was released with patchelf 0.10, but the verification still checks for 0.9:

auditwheel/auditwheel/patcher.py

Lines 41 to 45 in 686024e

	if m and tuple(int(x) for x in m.group(1).split('.')) >= (0, 9):
	return
	raise ValueError(('patchelf %s found. auditwheel repair requires '
	'patchelf >= 0.9.') %
	version)

Or maybe it is unrelated, was just wondering 🙂