Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when binaries from binary wheels are used #150

Closed
jooon opened this issue Apr 28, 2016 · 9 comments
Closed

Crash when binaries from binary wheels are used #150

jooon opened this issue Apr 28, 2016 · 9 comments

Comments

@jooon
Copy link

jooon commented Apr 28, 2016

On 2016-04-13 a big project to build binary wheels of everything for Linux on https://pypi.python.org was started.

See for instance: https://pypi.python.org/pypi/numpy

This is great. It should cut down build time, probably produce more stable and better optimize builds in general.

However this can lead to build failures on a normal dh_virtualenv build started like this:

%:
    dh $@ --with python-virtualenv

First it can crash on dh_strip:

   dh_link
   dh_compress
   dh_fixperms
   dh_strip
BFD: debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/stfcSdQH: Not enough room for program headers, try linking with -N
strip:debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/stfcSdQH[.gnu.hash]: Bad value
BFD: debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/stfcSdQH: Not enough room for program headers, try linking with -N
strip:debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/stfcSdQH: Bad value
dh_strip: strip --remove-section=.comment --remove-section=.note --strip-unneeded debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0 returned exit code 1
make: *** [binary] Error 29

Probably because the binutils package I have on Ubuntu 14.04 is different from the one used to build the binary package. Maybe some other reason as well.

If you manually exclude stripping those libraries with:

override_dh_strip:
    dh_strip --exclude=numpy --exclude=scipy

it will then fail in dh_shlipdeps:

   dh_makeshlibs
   dh_shlibdeps
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: error: couldn't find library libgfortran-ed201abd.so.3.0.0 needed by debian/«PROJECT»/lib/python2.7/site-packages/scipy/.libs/libopenblasp-r0-39a31c03.2.18.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libopenblasp-r0-ea933816.2.18.so'
dpkg-shlibdeps: error: couldn't find library libgfortran-ed201abd.so.3.0.0 needed by debian/«PROJECT»/lib/python2.7/site-packages/numpy/.libs/libopenblasp-r0-ea933816.2.18.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: error: cannot continue due to the errors listed above

which is probably only a problem finding internal libraries.

You can again work around by excluding:

override_dh_shlibdeps:
    dh_shlibdeps --exclude=numpy --exclude=scipy

I am not sure however if it is possible for dh_virtualenv to figure this out automatically.

@nailor
Copy link
Collaborator

nailor commented May 10, 2016

One option could be to skip the dh_shlibdeps altogether, but it would probably make many builds even worse, since at the moment that kind of works as a nice automation step.

@mmulqueen
Copy link

mmulqueen commented Jun 30, 2016

I've run into the same problem - sharing my experience in case someone turns up here via Google. Ultimately, after spending a lot of time mucking about with this trying to come up with a better way, I wound up using pip's --no-binary :all: option. I didn't want to disable dh_shlib/dh_shlibdeps selectively. I'm sure there's a way to use devpi with only locally built wheels (so they don't have to be compiled each time), but I ran out of time.

My debian/rules now looks more or less like this:

#!/usr/bin/make -f
override_dh_virtualenv:
    dh_virtualenv --extra-pip-arg='--no-binary' --extra-pip-arg=':all:'

%:
    dh $@ --with python-virtualenv

Edit: Also see this issue:
numpy/numpy#7570

@jhermann
Copy link
Contributor

jhermann commented Jul 1, 2016

You can likely shorten that to --extra-pip-arg='--no-binary=:all:'.

bochecha pushed a commit to ideascube/ideascube that referenced this issue Jul 26, 2016
We want, and need, to rebuild everything from source, without using
pre-built wheels.

One reason is that it's just the right thing to do, but it's especially
pressing because the build actually fails with wheel, as described here:

    spotify/dh-virtualenv#150

The solution would be to pass the --no-binary=:all: option to pip,
except it appeared in pip 7.0.0, and we're stuck on a much older one.

Fortunately, theold version had --no-use-wheel which does the same.

This commit bumps the dh-virtualenv dependency to 0.10 (only in the
Jessie backports), so we can use its --pip-extra-args option to tell pip
not to use wheels.

Relates to #408
@nailor
Copy link
Collaborator

nailor commented Aug 2, 2016

Since there is probably not much we can do here for dh-virtualenv, I'll close the issue. Feel free to reopen if you disagree!

@asottile
Copy link
Contributor

asottile commented Feb 2, 2017

I have a slightly better solution to this enabled by the recently added --preinstall option.

The problem here is manylinux wheels. You can conveniently disable them by installing https://github.com/asottile/no-manylinux1

This can be done with --preinstall=no-manylinux1

@asottile
Copy link
Contributor

asottile commented Feb 2, 2017

I should clarify. This allows you to avoid the problematic wheels while still enabling pure python wheels

@mdcroce
Copy link

mdcroce commented Mar 15, 2017

I'm getting an error when trying to build matplotlib, the solutions above didn't solve the problem.

subprocess.CalledProcessError: Command '['/home/matheus/linc-research-dev/projects/teste-core/debian/linc-teste-core/opt/venvs/linc-teste-core/bin/python', '/home/matheus/linc-research-dev/projects/teste-core/debian/linc-teste-core/opt/venvs/linc-teste-core/bin/pip', 'install', '--log=/tmp/tmpI5c4jV', '--no-binary=:all:', '-r', './requirements.txt']' returned non-zero exit status 1 debian/rules:7: recipe for target 'override_dh_virtualenv' failed
make[1]: *** [override_dh_virtualenv] Error 1
make[1]: Leaving directory '/home/matheus/linc-research-dev/projects/teste-core'
debian/rules:4: recipe for target 'binary' failed
make: *** [binary] Error 2

@jhermann
Copy link
Contributor

Appending stuff to closed tickets is not a good idea.

And likely you'd be better off to actually use wheels and then prevent dh_strip from choking on them. Make sure you actually have the build deps when using --no-binary, i.e. try his manually in a virtualenv and check the output.

@jhermann
Copy link
Contributor

Just for reference, the on-going discussion on this is #203 and the best solution (until that patchelf fix bubbles to the right places) is using override_dh_strip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants