Default OPENBLAS_DYNAMIC_ARCH to off, add one master "portable Julia" build flag

[tkelman]

This is a reminder for something I've been meaning to do for a while. Figured I'd open a tracking/discussion/up-for-grabs issue on it. For building Julia from source, my guess is that most people are only intending to run Julia on the single machine they're compiling it on. But building OpenBLAS with OPENBLAS_DYNAMIC_ARCH enabled, which is our current default, takes a really long time.

I think we should switch the default - but at the same time, for the sake of the people who are intending to run Julia on multiple machines, it would be convenient to add a single consolidated flag that sets OPENBLAS_DYNAMIC_ARCH and MARCH and whatever else makes sense when building a portable version of Julia, all at the same time. Sort of like we have the one master MKL flag that sets all the other related Intel library flags together. Name of the flag is subject to bikeshedding. Maybe JULIA_PORTABLE or something like that?

[StefanKarpinski]

IIRC, part of the reason for building OpenBLAS dynamic everywhere was so that more people would test the "standard" setup. Otherwise you end up in a situation where most developers build their own and are using one version of OpenBLAS while most users are not building their own and are using a different version of OpenBLAS. Seems like a recipe for finding bugs that only occur in released versions.

[tkelman]

OpenBLAS with dynamic arch enabled is already going down different code paths on different machines, we've found a fair share of only-on-specific-hardware bugs. So the runtime problem isn't any different, though with dynamic arch at least you can manually ask for older kernels now - asking for newer than your hardware doesn't work though (obviously). That leaves build-time issues, and I think with the level of automation that we have building nightlies now that might not be so much of a concern?

[ViralBShah]

There have been a couple of issues in the past that have surfaced only with DYNAMIC_ARCH. Most developers on base probably do not recompile from source all that often.

How about leave it on by default, but make it easy to turn off for those who do need to build frequently? The flag name could just be OPENBLAS_DYNAMIC_ARCH. The collection of portability flags could be JULIA_PORTABLE or BUILD_PORTABLE or something.

[tkelman]

I'm not so concerned with existing developers as newcomers. People who use the binaries, find a bug or missing feature, want to contribute a fix, but if they need to switch to a source build it takes a long time for the first build. I think there are (going to be) more people in this boat than existing developers, for whom OPENBLAS_DYNAMIC_ARCH already exists and is already easy to turn off. Maybe we just document it more clearly, hey you can save build time by setting this.

[ViralBShah]

I can buy that argument, and is something we can surely try out for a few weeks. Many of us will have DYNAMIC_ARCH set so that we are on the same setup as release.

I would also want to turn off all the tests that the libraries in deps run - fftw and gmp I think have some of the longest running tests, giving a poor first build experience.

[StefanKarpinski]

I'm not so concerned with existing developers as newcomers.

That's a convincing concern. Making the first-time build faster lowers the barrier to entry for people who want to contribute changes to Julia, which is a really important consideration.

[Keno]

bump :)

[ViralBShah]

I am ok with turning off DYNAMIC_ARCH in the developer build and enabling in the distribution builds.

[tkelman]

but that wouldn't be backported, so 0.5.x is not appropriate

[tkelman]

nice-to-have but not immediately release blocking IMO

[StefanKarpinski]

This seems like a feature and therefore not a 1.0-blocker.

[ViralBShah]

This really isn't an issue, given that LLVM takes way longer to build - for a newcomer. I suggest closing.

[ViralBShah]

Building openblas does not feel as painful as it used to once upon a time. Maybe we are used to it now, and also maybe because updates are a lot less frequent.

[tkelman]

openblas definitely takes longer to build than llvm. and with the function multiversioning features this would be more useful than just for openblas

[ViralBShah]

While it would be great to have the portable julia level flags, I don't think we should have the default build turn off dynamic_arch in openblas. Perhaps we can reopen the issue with a title to focus on portability rather than changing the defaults.

[tkelman]

Why not? Turning it off would save considerably on build time, and not be noticeable for the vast majority of what people do with source builds.