Julia package repository for CentOS/Fedora

[ViralBShah]

I wonder if there is some kind of build farm out there that makes it possible for us to generate Julia binaries for multiple linux distros. I believe work has already been done to build linux RPMs for CentOS/Fedora, and it would be great if we can have nightlies and releases through a yum repository just like we do for ubuntu right now.

[ViralBShah]

Cc: @nalimilan @staticfloat @yarapavan

[JeffBezanson]

What is the criterion for closing this issue? Distro package maintainers are working on this. What does "various" include?

[ViralBShah]

Ok, I have updated this issue to be specific to CentOS/Fedora.

[JeffBezanson]

Highly reminiscent of #2523

[ViralBShah]

Slightly different - this is not just about packaging, but setting up a repository so that users can easily install with dependencies and get nightlies, etc.

[ViralBShah]

https://www.centos.org/docs/5/html/5.2/Deployment_Guide/s1-yum-repo-setup.html

[nalimilan]

I've set up a Copr project, which offers a repository for Fedora 19 and 20, and RHEL/Centos 6 and 7:
http://copr.fedoraproject.org/coprs/nalimilan/julia/

At the moment it doesn't work for RHEL6 since we need to build a few more packages which are too old there. But it shouldn't be too hard to fix.

As regards building nightlies, I should also be quite easy by simply running an automated script on a custom server. If you can provide the server, I can write the script. For example, I can adapt this one: https://github.com/domcleal/puppet-nightly-builder/blob/master/build.sh

[tkelman]

@nalimilan I've been building myself on RHEL5 with only occasional trouble. Let me know if you need any assistance there.

[nalimilan]

@tkelman Good to hear. Though the harder part is to make separate packages, using where possible the system libraries. If RHEL5 packages are really too old for Julia, maybe bundling everything would be OK.

Anyway, Copr does not support RHEL5, so we will need to build the package elsewhere.

[ViralBShah]

I believe we should be able to do this on julia.mit.edu. It is an ubuntu box, and we can perhaps install vagrant. We also have a farm of a dozen old (~10 years) unused machines that are online that could be repurposed with any distro, but may not be worth the hassle.

Cc: @Keno

I believe CentOS 6 should be good starting point. We can do CentOS 5 if someone asks.

[tkelman]

Hopefully not too many people are still on RHEL/Centos 5, it's a pretty painful experience. I'll do my best to keep checking that things stay functional though (at the moment they aren't, see JuliaLang/libuv#26). Will become harder when we switch to LLVM 3.5, it's tough to get C++11 compilers in the repos for these distributions.

[crayxt]

In the meantime, why not provide binary builds for Linux just like you do for Windows/OSX?
Last week I've built julia on Centos6, and then this build passed tests on Debian sid and Fedora 20 too.

[crayxt]

About RHEL6, OpenBLAS has compile issues on RHEL6 due to outdated binutils. The build fails for me on HASWELL kernel.

[nalimilan]

@ViralBShah You don't need a powerful machine, the script is just downloading the latest Julia, making an RPM package and asking the Copr server to run the builds. But you need to be able to create a source RPM, so it would be easier to use a RPM distro. Really, the only other requirement is that the machine is online at least once a day.

[ViralBShah]

Just build OpenBLAS for NEHALEM on centos 6. The compiler is too old to support the new instructions.

Binaries for Linux are a good idea. A tar.gz built on an oldish distro should do the trick for most users.

[staticfloat]

@nalimilan you can talk to me about servers once you're ready. One of my projects this summer is to create a small virtual machine buildfarm using Vagrant to allow us to build binaries for everything, all in one place. I don't have the automated vagrant stuffs working yet, (That won't happen until mid-July, once I'm over in Cambridge) but these kinds of issues are definitely what we'll be hoping to address with the infrastructure I'll be building.

[nalimilan]

@staticfloat OK, will do.

[crayxt]

@ViralBShah @nalimilan It turns out, Scientific Linux provides rebuilt version of RHEL devtoolset-2, using which I was able to build openblas with dynamic arch. This devtoolset includes more recent gcc and binutils.
devtoolset's link is http://ftp.scientificlinux.org/linux/scientific/6/x86_64/external_products/devtoolset/2

the version of julia built with this toolset passed all tests on regular Centos6, so I think no extra libraries are required on end user machines.

[crayxt]

@ViralBShah I succeeded in building julia binaries on Centos5 using the devtoolset from Scientific Linux. Tests pass on Centos 5, 6, 7. Builds are done in chroot, using mock tool.
This build requires gfortran libs to be installed (Centos 5: libgfortran44). I can share configs - nothing special though.

So you can lower entry point to Centos 5.

[ViralBShah]

Thanks @crayxt. Can you provide a Make.user.centos5?

[crayxt]

@ViralBShah I only set JULIA_CPU_TARGET=core2
Here is the gist with setup info https://gist.github.com/crayxt/90aabcaab2c725982624

Although Scientific Linux states that their rebuilt version of RedHat devtoolset is provided in testing purposes only (http://linux.web.cern.ch/linux/devtoolset/).

[nalimilan]

@crayxt That's great you got so far on RHEL/CentOS. I copied your packages to make Julia build on EPEL7. But do you know why it fails on your Copr for EPEL6? Have you tried building a more recent LAPACK?

http://copr-be.cloud.fedoraproject.org/results/baurzhanm/julia-epel6/epel-6-x86_64/julia-0.3.0-5.el6/build.log

[crayxt]

@nalimilan I am glad I can help. Pay attention when copying them, I cannot guarantee their correctness. I did not try to build more recent lapack.

[nalimilan]

@crayxt Sure. It would be great if you could add a changelog entry stating what you modified in the packages.

@ViralBShah @staticfloat I've put together a script to build RPM nightlies, based on the script for Debian. It works on my machine, but at the moment it needs my credentials to put the SRPM on a FTP server, and then to tell the Copr build service to build it. Regarding the first point, we should find another public place to put the tarballs; regarding the second one, I think we should set up a Copr repo for the Julia team. Should be fairly easy to get working.
The script is at: http://nalimilan.perso.neuf.fr/transfert/julia-nightlies.sh
(I've commented out some parts which were not needed for a first attempt.)

[tkelman]

@nalimilan I've been using sourceforge as a dumping ground for binaries, it's a little slow and antiquated, but it works for my purposes. Should be able to upload files in a script over scp.

[crayxt]

@nalimilan I will - just give me the list of packages that you copied.

[nalimilan]

@tkelman Yeah, any place will do.

@crayxt Everything. ;-) gcc48, fftw, openblas.

[crayxt]

@nalimilan I guess it is because of the way openblas is packaged in EPEL6, see, in my full chroot build with make dist I have

[baur@f0 julia]$ nm -D libopenblas.so | grep sgeqrt
0000000001c835d0 T LAPACKE_sgeqrt
0000000001c836e0 T LAPACKE_sgeqrt2
0000000001c83790 T LAPACKE_sgeqrt2_work
0000000001c83a90 T LAPACKE_sgeqrt3
0000000001c83b40 T LAPACKE_sgeqrt3_work
0000000001c83e40 T LAPACKE_sgeqrt_work
0000000001752fb0 T sgeqrt_
0000000001753380 T sgeqrt2_
00000000017539b0 T sgeqrt3_

While in library provided in your repo

nm -D ./usr/lib64/libopenblasp.so.0 | grep sgeqrt

gives nothing.
In spec file of Fedora's openblas spec file http://pkgs.fedoraproject.org/cgit/openblas.git/tree/openblas.spec I see the following switch

# Do we have LAPACKE? (Needs at least lapack 3.4.0)
%if 0%{?fedora} > 16
%global lapacke 1
%else
%global lapacke 0
%endif

RHEL7, AFAIK is based on Fedora 18.

[crayxt]

@nalimilan I've enabled tests in my spec file and adapted it to epel5, 6 and 7. Only 64-bits can be built at the moment https://copr.fedoraproject.org/coprs/baurzhanm/devtoolset-rebuild/build/51366/

[nalimilan]

Good catch. Unfortunately, I've tried building Julia with the reference LAPACK, and still no luck:
http://copr-be.cloud.fedoraproject.org/results/nalimilan/julia/epel-6-x86_64/julia-0.3.1-2.fc20/build.log I need to do more investigation.

Regarding your package, wouldn't it be better to install files to /usr/local/ instead of /opt? It seems that $PATH generally includes the former, but not the latter. That way, people could just type julia without wondering where Julia was installed, and it would be more consistent if we offer your package and my package in the same Copr.

[nalimilan]

Ah, got it, sgeqrt was added in LAPACK 3.4. I'll add the list of minimum versions to README.md when I'm done with that stuff.

[crayxt]

@nalimilan I've added profile scripts to my package, they are sourced when shell starts and PATH gets updated. For sh shells (/etc/profile.d/julia.sh) it looks like this

JULIA_DIR=/opt/julia
if ! echo ${PATH} | /bin/grep -q $JULIA_DIR/bin ; then
   PATH=$JULIA_DIR/bin:${PATH}
fi
export PATH

See https://copr.fedoraproject.org/coprs/baurzhanm/devtoolset-rebuild/build/51451/

[ViralBShah]

That's right - a newer lapack is nice to have, since there are some nice performance improvements in there.

[nalimilan]

@crayxt OK, fine.

Regarding my EPEL6 package, the issue with LAPACK is fixed. Now 64-bit build passes, but there's a failure with sin on 32-bit (note I'm using openlibm):

/builddir/build/BUILD/julia-0.3.1/build/usr/bin/julia --check-bounds=yes -f ./runtests.jl math
     �[1m*�[0m �[31mmath�[0m
exception on 1: ERROR: assertion failed: |sinpi(convert(T,x))::T - convert(T,sin(pi * x))| <= 9.536743e-7
  sinpi(convert(T,x))::T = -0.8090169
  convert(T,sin(pi * x)) = -1.0
  difference = 0.19098312 > 9.536743e-7
while loading math.jl, in expression starting on line 8
ERROR: assertion failed: |sinpi(convert(T,x))::T - convert(T,sin(pi * x))| <= 9.536743e-7
  sinpi(convert(T,x))::T = -0.8090169
  convert(T,sin(pi * x)) = -1.0
  difference = 0.19098312 > 9.536743e-7
while loading math.jl, in expression starting on line 8
while loading /builddir/build/BUILD/julia-0.3.1/test/runtests.jl, in expression starting on line 35
make: *** [math] Error 1

http://copr-be.cloud.fedoraproject.org/results/nalimilan/julia/epel-6-i386/julia-0.3.1-2.fc20/build.log

[ViralBShah]

I think this is the same as an already known openblas issue:

JuliaMath/openlibm#57

[nalimilan]

Indeed. So it would be a problem when compiling openlibm. I'll try rebuilding it with a more recent compiler, and setting MARCH to something like pentium4.

[ViralBShah]

Another thing to check is USE_SYSTEM_LIBM=1.

[nalimilan]

Another thing to check is USE_SYSTEM_LIBM=1.

What do you mean, enabling it? For now I'm setting USE_SYSTEM_LIBM=0.

[ViralBShah]

Yes, I am wondering if the problem goes away if you use the system libm.

[nalimilan]

Yes, it passes with USE_SYSTEM_LIBM=1, but fails a bit later when testing gamma. I'm now trying to build openlibm with a newer gcc to see if it makes any difference.

[nalimilan]

After rebuilding both openlibm and openspecfun with gcc 4.8.2, the Julia RPM for EPEL6 works! Now let's try EPEL5...

[ViralBShah]

Wow, this is a major milestone!

[nalimilan]

OK, now the Julia 0.3.1 RPM builds on EPEL5 64-bit, with only limited modifications. Most of the work consisted in porting the required dependencies. EPEL5 32-bit does not work because of build failures of llvm and double-conversion, not sure it's worth investigating.

Any testing of EPEL5 and 6 packages would be appreciated. My main concern is whether enabling the Copr won't pull dependencies which may conflict with distribution packages. Normally it should be OK as I made several packages parallel-installable when they are not compatible with existing versions, but I haven't tested that yet.

[ViralBShah]

So, from here, what does it take for us to be part of the next Fedora release? I know the Debian world's process, but not the RedHat world.

[crayxt]

Julia is already accepted to current Fedora release (Fedora 20), thanks to @nalimilan.
See https://bugzilla.redhat.com/show_bug.cgi?id=1040517
Being part of EPEL5 and 6 is rather complicated due to bunch of dependencies which are too old there.

[ViralBShah]

Awesome!

[ViralBShah]

It seems like we can close this issue then.

[tkelman]

@nalimilan do you have instructions somewhere for how to add your EPEL5 repo? I agree that 32 bit RHEL5 is a low-value target, as long as more recent versions can work on 32 bit. Generic binaries for 32 bit Linux might be able to work on RHEL5, we'll see when someone tries making a set.

One thing I'm keeping an eye out for is upgrading our LLVM version will require using a C++11 compiler, which can be a bit tough to get on RHEL5 or 6 without the devtoolset. If the packages can be built with the devtoolset without requiring the user to have the devtoolset to run them, then maybe it's a non-issue.

[staticfloat]

I made some progress towards 32bit generic binaries today. The generic
binaries are built on CentOS 5.
On Oct 11, 2014 9:38 PM, "Tony Kelman" [email protected] wrote:

@nalimilan https://github.com/nalimilan do you have instructions
somewhere for how to add your EPEL5 repo? I agree that 32 bit RHEL5 is a
low-value target, as long as more recent versions can work on 32 bit.
Generic binaries for 32 bit Linux might be able to work on RHEL5, we'll see
when someone tries making a set.

One thing I'm keeping an eye out for is upgrading our LLVM version will
require using a C++11 compiler, which can be a bit tough to get on RHEL5 or
6 without the devtoolset. If the packages can be built with the devtoolset
without requiring the user to have the devtoolset to run them, then maybe
it's a non-issue.

—
Reply to this email directly or view it on GitHub
#7363 (comment).

[crayxt]

@tkelman On the other hand I have rpms of Julia with all libs bundled.
https://copr.fedoraproject.org/coprs/baurzhanm/devtoolset-rebuild/build/52446/
On Oct 12, 2014 11:07 AM, "Elliot Saba" [email protected] wrote:

I made some progress towards 32bit generic binaries today. The generic
binaries are built on CentOS 5.
On Oct 11, 2014 9:38 PM, "Tony Kelman" [email protected] wrote:

@nalimilan https://github.com/nalimilan do you have instructions
somewhere for how to add your EPEL5 repo? I agree that 32 bit RHEL5 is a
low-value target, as long as more recent versions can work on 32 bit.
Generic binaries for 32 bit Linux might be able to work on RHEL5, we'll
see
when someone tries making a set.

One thing I'm keeping an eye out for is upgrading our LLVM version will
require using a C++11 compiler, which can be a bit tough to get on RHEL5
or
6 without the devtoolset. If the packages can be built with the
devtoolset
without requiring the user to have the devtoolset to run them, then
maybe
it's a non-issue.

—
Reply to this email directly or view it on GitHub
#7363 (comment).

—
Reply to this email directly or view it on GitHub
#7363 (comment).

[nalimilan]

@tkelman I've just updated instructions on the Copr: https://copr.fedoraproject.org/coprs/nalimilan/julia/ Regarding the devtoolset, I'm already using it anyway for several packages, including Julia, and apparently that doesn't force installing it on the user's system.

So now we actually offer three install options to users of RHEL/Centos 5!

[nalimilan]

@staticfloat Do you think you could add the Copr nightly build status to http://status.julialang.org/? The return value of copr-cli should tell you whether the build succeeded (though at the moment it always fails on EPEL5, but that's going to be fixed soon).

[staticfloat]

Sure, I'll rename SRPM to Copr or something, and increase the timeout so that we can wait until the copr build finishes.

[staticfloat]

I think at this point, we can close this issue. Good work, everyone.