GCC multi-versioning

[staticfloat]

I'd like to try out the crazy idea of building a couple of different GCC versions and creating e.g. x86_64-linux-gnu-gcc6.4.0 and x86_64-linux-gnu-gcc7.3.0 and x86_64-linux-gnu-gcc8.1.0 shards; this would allow us to solve the inevitable problems stemming from:

• I built Julia from source and used GCC 5.4 or 8.1, so all your BinaryBuilder nonsense is useless to me, since you use GCC 7.3.0.

• I am using Julia 2.0 which uses GCC 8.1, but LibFoo.jl wants to support both Julia 0.7 and 2.0.

[ViralBShah]

Won't that mean that we will have a combinatorial explosion of binaries for every compiler version? Seems like we should instead restrict the officially supported list of compilers and request people to use official binaries if they don't have the right compiler.

[staticfloat]

Well, the problem is that we're doing that right now and we're not able to use BinaryBuilder on some prominent packages such as Ipopt.jl, because apparently Ipopt.jl always requires users to compile something on their Linux machines due to legal reasons (there's some proprietary optimizer that everyone uses because it's much faster/better than the opensource optimizers, and it's not allowed to be distributed which means we can't just build it using BB). This means that, since most users don't have either an older or newer version of gfortran (and since the SOVERSION of gfortran has gone from 3 -> 4 -> 5 in the last year) we can't use BinaryBuilder with Ipopt.jl on Linux at all. We could avoid this if we had a version of IpoptBuilder that built against GCC 5.4 (as the latest GCC version that used libgfortran.so.3), GC 7.3.0 (as the latest GCC version that used libgfortran.so.4) and GCC 8.1.0 (as the latest GCC version that uses libgfortran.so.5). So yes, this would ~triple the number of builds we'd have to do.

[ViralBShah]

That seems like a huge hammer, if that is the only reason. But would you only do that for libraries like Ipopt? I guess it would be the package author's decision.

[Keno]

there's some proprietary optimizer that everyone uses because it's much faster/better than the opensource optimizers, and it's not allowed to be distributed which means we can't just build it using BB

Link? I'd like to make sure this isn't just a misunderstanding of some legalese.

[ViralBShah]

cc @mlubin

[mlubin]

There are two separate issues.

• Ipopt supports multiple backends for sparse linear algebra. MUMPS is the slowest and the only one that's available under an open-source license. Advanced users would like to compile their own version of Ipopt that links to commercial linear algebra libraries like MA57, so this means that Ipopt.jl needs to support user-compiled Ipopt. Otherwise this is orthogonal to BinaryBuilder. BinaryBuilder should and can only distribute Ipopt with MUMPS.

• The gfortran dependency issue means that the BB binaries are incompatible with lots of versions of Julia out in the wild, especially on Linux where it's more likely that users have compiled from source. We didn't switch to the BB binaries on Linux in order to avoid breaking things and facing the user complaints. We switched on Mac OS because the homebrew binaries were not working at all, and on Windows because why not. BB on Mac OS and Windows seems to be working great so far.

[stevengj]

Advanced users would like to compile their own version of Ipopt that links to commercial linear algebra libraries like MA57,

The ideal thing would be to patch Ipopt so that it doesn't need to be recompiled to do this. Is there any chance that they could dlopen the commercial libraries if some configuration is set?

[staticfloat]

The ideal thing would be to patch Ipopt so that it doesn't need to be recompiled to do this. Is there any chance that they could dlopen the commercial libraries if some configuration is set?

While that is true; this doesn't change the fact that we will need to have some kind of multi-GCC support in BinaryBuilder, if only to support different Julia versions.

[mlubin]

The ideal thing would be to patch Ipopt so that it doesn't need to be recompiled to do this. Is there any chance that they could dlopen the commercial libraries if some configuration is set?

Edit: I wasn't aware, but this is possible already. I cannot test this personally because I don't have access to MA57, but useful to keep in mind when it comes up again.

[juan-pablo-vielma]

I tested the HSL dynamic loading @mlubin mentioned and it seems to work ok. I used this experimental version of Ipopt.jl that uses BB/BP to provide binary dependencies that were not compiled with the HSL libraries.

Seems to be working in julia v0.6 for x86-64 linux, macos and win10 for ma27, ma57, ma77, ma86 and ma97. Just need to drop libhsl (and its dependencies) to LD_LIBRARY_PATH/DYLD_LIBRARY_PATH/PATH and pass the appropriate parameter to Ipopt (e.g. IpoptSolver(print_level=5,linear_solver="ma57")

My test was running Ipopt.jl tests for all linear solvers and all but ma77 pass everything (and Ipopt reports its using the libraries. e.g. This is Ipopt version 3.12.10, running with linear solver ma97).

One interesting thing is that Ipopt and HSL seem to be using different versions of gfortran:
Ipopt uses julia version and HSL used the whatever was used to compile it (see below). Could this cause trouble?

e.g. in linux:

jvielma@chewie:~$ ldd ~/lib/libhsl.so 
	linux-vdso.so.1 =>  (0x00007ffd14bd1000)
	libcoinlapack.so.0 => /home/jvielma/lib/libcoinlapack.so.0 (0x00007f0fed964000)
	libcoinblas.so.0 => /home/jvielma/lib/libcoinblas.so.0 (0x00007f0fed73d000)
	libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f0fed412000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0fed109000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0fecd3f000)
	libgcc_s.so.1 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libgcc_s.so.1 (0x00007f0fecb27000)
	libquadmath.so.0 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libquadmath.so.0 (0x00007f0fec8e7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0fedf48000)
jvielma@chewie:~$ ldd ~/.julia/v0.6/Ipopt/deps/usr/lib/libipopt.so
	linux-vdso.so.1 =>  (0x00007ffc2ddde000)
	libcoinmumps.so.1 => /home/jvielma/.julia/v0.6/Ipopt/deps/usr/lib/libcoinmumps.so.1 (0x00007f1b8dd2d000)
	libcoinlapack.so.1 => /home/jvielma/.julia/v0.6/Ipopt/deps/usr/lib/libcoinlapack.so.1 (0x00007f1b8da65000)
	libcoinblas.so.1 => /home/jvielma/.julia/v0.6/Ipopt/deps/usr/lib/libcoinblas.so.1 (0x00007f1b8d83d000)
	libgfortran.so.4 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libgfortran.so.4 (0x00007f1b8d46b000)
	libquadmath.so.0 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libquadmath.so.0 (0x00007f1b8d22b000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1b8d027000)
	libstdc++.so.6 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libstdc++.so.6 (0x00007f1b8cca9000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1b8c9a0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1b8c5d6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1b8e5ef000)
	libgcc_s.so.1 => /home/jvielma/julia-9d11f62bcb/bin/../lib/julia/libgcc_s.so.1 (0x00007f1b8c3be000)
	libcoinmetis.so.1 => /home/jvielma/.julia/v0.6/Ipopt/deps/usr/lib/libcoinmetis.so.1 (0x00007f1b8c16e000)
jvielma@chewie:~$ echo $LD_LIBRARY_PATH 
/home/jvielma/lib:/home/jvielma/julia-9d11f62bcb/bin/../lib/julia

and macos:

han:~ jvielma$ otool -L ~/lib/libhsl.dylib
/Users/jvielma/lib/libhsl.dylib:
	/Users/jvielma/delme/lib/libcoinhsl.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
	/usr/local/opt/gcc/lib/gcc/8/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
han:~ jvielma$ otool -L ~/.julia/v0.6/Ipopt/deps/usr/lib/libipopt.dylib 
/Users/jvielma/.julia/v0.6/Ipopt/deps/usr/lib/libipopt.dylib:
	/workspace/destdir/lib/libipopt.1.dylib (compatibility version 12.0.0, current version 12.10.0)
	@rpath/libcoinmumps.1.dylib (compatibility version 8.0.0, current version 8.0.0)
	@rpath/libcoinmetis.1.dylib (compatibility version 5.0.0, current version 5.5.0)
	@rpath/libcoinlapack.1.dylib (compatibility version 7.0.0, current version 7.6.0)
	@rpath/libcoinblas.1.dylib (compatibility version 6.0.0, current version 6.6.0)
	@rpath/libgfortran.4.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
	@rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
han:~ jvielma$ echo $DYLD_LIBRARY_PATH 
/Users/jvielma/lib/:/Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia

[staticfloat]

Crazy idea: implemented.