Documenting best practice for profiling *internals* of Julia

[ArchRobison]

Julia compilation speed is important, and to study it requires profiling the compiler, including LLVM. So I'd like to use this issue to collect the best known practices, and maybe even get it documented somewhere. Basically, the issue is how to turn on debugging symbols in the binaries without turning on assertions and other baggage.

[Updated 2014-Apr-28] See later note for current best practice. Rest of this note retained only for preserving context of later comments.

---

Here is what I've been doing so far (which is likely not best possible practice):
1.In Make.user, include the lines:

USE_INTEL_JITEVENTS = 1

The first line is necessary if using Intel VTune Amplifier.
2. Start a release build. After LLVM is configured, I stopped the process and edited julia/deps/llvm-3.3/build_Release and uncommented these two lines (not consecutive):

DEBUG_SYMBOLS = 1
KEEP_SYMBOLS = 1

Then I rebuilt LLVM and Julia.

It appears that the LLVM configure script has options --enable-debug-symbols and --enable-keep-symbols to set said symbols above. But it's unclear to me if these can be somehow set in Make.user and passed on down to LLVM's configure script.

[tkelman]

For better ways of sending new flags to LLVM's configure:
If you're not using LLVM_SANITIZE in Make.user then you should be able to use LLVM_CC = --enable-debug-symbols --enable-keep-symbols to get them sent to LLVM's configure (not the intended usage here, could easily change). The more proper way to do it would be adding a couple lines here https://github.com/JuliaLang/julia/blob/master/deps/Makefile#L204-L246 to put these into LLVM_FLAGS based on some new Make.user parameter. Or allowing LLVM_FLAGS to start out nonempty by using += from the beginning.

[ViralBShah]

You can now set LLVM configure options in Make.user. Also, make LLVM_DEBUG=1 should do what you want.

[ivarne]

Shold we document all these Make.user configuration options somewhere?

[timholy]

The FAQ?

[ViralBShah]

The thing is that you can pretty much override anything in our Makefiles through Make.user. If we want to do this, we should probably just have a Building Julia section in our manual, and thoroughly document various build related configuration options.

[ivarne]

There is of course the problem that build options tend to be added without much thought (or inadvertently broken), so any documentation will quickly be outdated. Still I think some of these options deserves to listed somewhere. Makefiles can be quite hard to read if you want to find out how things work.

[ViralBShah]

I think the reverse ought to be true though. Options that are documented should be treated no different than APIs. Any changes to documented options should reflect in documentation and version updates.

[ArchRobison]

My impression is that LLVM_DEBUG=1 not only retains debugging symbols, but also turns off optimization. So it's not quite the right thing for doing profiling, since LLVM is in C++ and turning off inlining can severely skew profiles.

[ViralBShah]

We can certainly enable optimizations in LLVM_DEBUG, if ok with others. @loladiro ?

[Keno]

At some levels there's not much difference between retaining debug symbols and turning off optimizations. C++ relies on heavy inlining to make anything fast, so you'll never get great stack traces when debugging optimized code.

[ViralBShah]

I guess there is the generic LLVM_FLAGS, which you can use to pass in the exact sequence of configure options you want. Perhaps we just leave it at that?

[ArchRobison]

Using LLVM_FLAGS seems reasonable to me. With #e220dd2, here's current best practice that I know of for compiling Julia for internal profiling:

1. In Make.user, include the lines:

USE_INTEL_JITEVENTS = 1
LLVM_FLAGS += --enable-debug-symbols --enable-keep-symbols
SHIPFLAGS += -g

[Updated 2014-May-7] The first line is necessary if you are using Intel VTune Amplifier. If using Intel VTune Amplifier, consider applying this patch, otherwise you may see warnings about the "sigalstack is too small".

1. Run make

Amplifier has been complaining about needing a bigger signal stack, but I haven't figured out whether that's hurting the profiles.

[vtjnash]

we have better support for this now (with make LLVM_DEBUG=symbols USE_BINARYBUILDER_LLVM=0)