root: separateDebugInfo = true by ShamrockLee · Pull Request #216864 · NixOS/nixpkgs

ShamrockLee · 2023-02-17T20:33:43Z

Description of changes

Build CERN ROOT with separateDebugInfo = true for easier debugging.

As a data analysis framework, users spend most of the time debugging their C++ analysis code, and errors like Segmentation Violation occur in various situations ranging from going out of bound to file missing due to network error. A call-stack error message with line numbers provides real help.

Things done

veprbl · 2023-02-17T22:13:54Z

Makes sense, but how much closure space do we have to pay for this?

ShamrockLee · 2023-02-17T22:20:41Z

The package size and the closure size grow smaller, which is somewhat strange.

nixpkgs_timeshift on  root-debug-info [$] took 4s
❯ nix path-info -S .#root
/nix/store/ddvxws306q93i377l1s77hfy59hxnj6z-root-6.26.10	1616296880

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -S .?ref=origin/master#root
/nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10	1623110720

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -s .#root
/nix/store/ddvxws306q93i377l1s77hfy59hxnj6z-root-6.26.10	 438663432

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -s .?ref=origin/master#root
/nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10	 445477272

Update: Just compared the outputs with Meld, and the files are all there, plus two new files:

/nix/store/ddvxws306q93i377l1s77hfy59hxnj6z-root-6.26.10/lib/libCore.so-gdb.py
/nix/store/ddvxws306q93i377l1s77hfy59hxnj6z-root-6.26.10/lib/libRooFitCore.so-gdb.py

ShamrockLee · 2023-02-17T22:41:13Z

Result of nixpkgs-review run on x86_64-linux 1

1 package failed to build:

python311Packages.hepmc3

5 packages built:

hepmc3
python310Packages.hepmc3
root
xfitter
yoda-with-root

ShamrockLee · 2023-02-19T17:58:34Z

I'm afraid that the debug information has got stripped away. I saw the strip running during the fixUp phase from OfBorg's build log.

Should we use dontStrip to prevent stripping as a whole or set separateDebugInfo = true and add the "debug" output to store the debug information separately?

veprbl · 2023-02-19T22:17:50Z

The separateDebugInfo wouldn't work for builtin stack traces?

ShamrockLee · 2023-02-20T15:42:15Z

The separateDebugInfo wouldn't work for builtin stack traces?

At least I don't know how to make it work.

ShamrockLee · 2023-02-20T15:45:40Z

The debug information occupies ~500MB when separated, but we get ~1.5GB size increasing when not splitted. The size of the ROOT derivation without the debug information is ~500MB.

The size increasing directly reflects on the closure size. It simply doubles (from 1.6GB to 3.1GB) after dontStrip = true;.

nixpkgs_timeshift on  root-debug-info [$!]
❯ nix path-info -s .#root
warning: Git tree '/run/media/root/data-btrfs/shamrock-shared/targets/Projects/NixOS/nixpkgs_timeshift' is dirty
/nix/store/gvmgba5jbaqk5fqdybwnl9cql4nnydih-root-6.26.10	1909723000

nixpkgs_timeshift on  root-debug-info [$!] took 3s
❯ nix path-info -S .#root
warning: Git tree '/run/media/root/data-btrfs/shamrock-shared/targets/Projects/NixOS/nixpkgs_timeshift' is dirty
/nix/store/gvmgba5jbaqk5fqdybwnl9cql4nnydih-root-6.26.10	3132206512

nixpkgs_timeshift on  root-debug-info [$!] took 2s
❯ nix path-info -s .?ref=origin/master#root
/nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10	 445477272

nixpkgs_timeshift on  root-debug-info [$!]
❯ nix path-info -S .?ref=origin/master#root
/nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10	1623110720

veprbl · 2023-02-20T18:25:45Z

The 2-4x increase would be hard to justify. Are you dealing with a large rebuild when setting up for a debug?

ShamrockLee · 2023-02-20T19:06:58Z

Just didn't know it's that large. I had thought it would be trivial 😓

I'll ask in the ROOT forum if there's a way to use the separated debug information, as it would hopefully not increase the closure size of root.out and other packages depending on it.

BTW, considering the huge size difference, is there something other than debug symbols that is normally stripped away?

Update: Here's the post in the ROOT Forum: https://root-forum.cern.ch/t/any-way-to-have-roots-builtin-stack-trace-use-seperated-debug-information/53702

veprbl · 2023-02-22T04:13:45Z

If you provide path to separated debug info to the ROOT, that would be a reference, so the closure size won't improve.

ShamrockLee · 2023-02-22T12:54:27Z

The closure size of the root.out output built with separateDebugInfo = true shrinks 7MB. Not sure about the optimization level. The shrink of the build may result from the change of optimization strategies. RelWithDebInfo is still a kind of "release" build anyway.

The reference could be made available either through a wrapper package, say, root-debug, or with user-side configuration using home-manager configuration or nixseparatedebuginfod plus corresponding .rc settings. Users who just install the root package should experience no closure size increasing.

A ROOT developer answered in the above forum post that ROOT uses gdb to generate the stack trace. Maybe there's a way to make it show the line number in the stack trace.

Closure size comparison

``` nixpkgs_timeshift (94a5042) [$] ❯ nix path-info -s .?ref=origin/master#root /nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10 445477272

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -S .?ref=origin/master#root
/nix/store/yscknw8l89m6xv46j4ff0mr7786fv22m-root-6.26.10 1623110720

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -s .#root
/nix/store/czl2bn94lbx01a1wamvyjgns0a90a7vc-root-6.26.10 438673256

nixpkgs_timeshift on  root-debug-info [$]
❯ nix path-info -S .#root
/nix/store/czl2bn94lbx01a1wamvyjgns0a90a7vc-root-6.26.10 1616306704

</details>

ShamrockLee · 2023-02-23T08:03:31Z

Ah! Here it is:


$ NIX_DEBUG_INFO_DIRS="$(nix path-info .#root.debug)/lib/debug" nix run .#root -- -l
root [0] TFile::Open("nonexistence.root")->GetName();
Error in <TFile::TFile>: file /run/media/root/data-btrfs/shamrock-shared/targets/Projects/NixOS/nixpkgs_timeshift/nonexistence.root does not exist
#0  0x00007f0ea23a89ea in wait4 () from /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6
#1  0x00007f0ea231a02b in do_system () from /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6
#2  0x00007f0ea2ae8815 in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x5cd800) at /build/root-6.26.10/core/unix/src/TUnixSystem.cxx:2104
#3  TUnixSystem::StackTrace (this=0x5cd800) at /build/root-6.26.10/core/unix/src/TUnixSystem.cxx:2395
#4  0x00007f0ea0752448 in cling::MultiplexInterpreterCallbacks::PrintStackTrace (this=<optimized out>) at /build/root-6.26.10/interpreter/cling/lib/Interpreter/MultiplexInterpreterCallbacks.h:158
#5  0x00007f0ea0751cee in cling_runtime_internal_throwIfInvalidPointer (Interp=0x65f3f0, Expr=0x1d49a70, Arg=<optimized out>) at /nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/include/c++/12.2.0/bits/unique_ptr.h:191
#6  0x00007f0e9b24d083 in ?? ()
#7  0x000000009b24d000 in ?? ()
#8  0x000000000134ea28 in ?? ()
#9  0x000000000134ea30 in ?? ()
#10 0x00007f0ea0751ca0 in ?? () from /nix/store/czl2bn94lbx01a1wamvyjgns0a90a7vc-root-6.26.10/lib/libCling.so
#11 0x000000000065f3f0 in ?? ()
#12 0x00007f0ea1e27890 in ?? () at /build/root-6.26.10/io/io/src/TFile.cxx:4352 from /nix/store/czl2bn94lbx01a1wamvyjgns0a90a7vc-root-6.26.10/lib/libRIO.so
#13 0x00007ffeb30587d0 in ?? ()
#14 0x00007f0ea07be3b8 in cling::IncrementalExecutor::executeWrapper (this=<optimized out>, function=..., returnValue=returnValue
entry=0x7ffeb30587d0) at /build/root-6.26.10/interpreter/cling/lib/Interpreter/IncrementalExecutor.cpp:381
#15 0x00007f0ea0753e21 in cling::Interpreter::RunFunction (this=0x65f3f0, FD=<optimized out>, res=0x7ffeb30587d0) at /nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/include/c++/12.2.0/bits/basic_string.h:1070
#16 0x00007f0ea0757981 in cling::Interpreter::EvaluateInternal (this=0x65f3f0, input=..., CO=..., V=0x7ffeb30587d0, wrapPoint=<optimized out>) at /build/root-6.26.10/interpreter/cling/lib/Interpreter/Interpreter.cpp:1414
#17 0x00007f0ea0757bb2 in cling::Interpreter::process (this=0x65f3f0, input=..., V=V
entry=0x7ffeb30587d0, T=T
entry=0x0, disableValuePrinting=disableValuePrinting
entry=false) at /build/root-6.26.10/interpreter/cling/lib/Interpreter/Interpreter.cpp:840
#18 0x00007f0ea08021ef in cling::MetaProcessor::process (this=this
entry=0x11d3080, input_line=..., compRes=
0x7ffeb30587bc: cling::Interpreter::kSuccess, result=result
entry=0x7ffeb30587d0, disableValuePrinting=disableValuePrinting
entry=false) at /build/root-6.26.10/interpreter/cling/lib/MetaProcessor/MetaProcessor.cpp:341
#19 0x00007f0ea068c00c in HandleInterpreterException (metaProcessor=0x11d3080, input_line=0x1d438b0 "#line 1 \"ROOT_prompt_0\"\nTFile::Open(\"nonexistence.root\")->GetName();", compRes=
0x7ffeb30587bc: cling::Interpreter::kSuccess, result=0x7ffeb30587d0) at /build/root-6.26.10/core/metacling/src/TCling.cxx:2441
#20 0x00007f0ea06a6528 in TCling::ProcessLine (this=0x65e730, line=<optimized out>, error=0x7ffeb3058b7c) at /nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/include/c++/12.2.0/bits/unique_ptr.h:191
#21 0x00007f0ea2c8a586 in TRint::ProcessLineNr (this=0x624e70, filestem=0x7f0ea2c9c298 "ROOT_prompt_", line=0x1d43710 "TFile::Open(\"nonexistence.root\")->GetName();", error=0x7ffeb3058b7c) at /build/root-6.26.10/core/rint/src/TRint.cxx:792
#22 0x00007f0ea2c8a97b in TRint::HandleTermInput (this=0x624e70) at /build/root-6.26.10/core/rint/src/TRint.cxx:650
#23 0x00007f0ea2ae4c66 in TUnixSystem::CheckDescriptors (this=this
entry=0x5cd800) at /build/root-6.26.10/core/unix/src/TUnixSystem.cxx:1306
#24 0x00007f0ea2ae7f38 in TUnixSystem::DispatchOneEvent (this=0x5cd800, pendingOnly=<optimized out>) at /build/root-6.26.10/core/unix/src/TUnixSystem.cxx:1061
#25 0x00007f0ea2a082e9 in TSystem::InnerLoop (this=0x5cd800) at /build/root-6.26.10/core/base/src/TSystem.cxx:406
#26 TSystem::Run (this=0x5cd800) at /build/root-6.26.10/core/base/src/TSystem.cxx:356
#27 0x00007f0ea299de73 in TApplication::Run (this=this
entry=0x624e70, retrn=retrn
entry=false) at /build/root-6.26.10/core/base/src/TApplication.cxx:1609
#28 0x00007f0ea2c8c1d0 in TRint::Run (this=this
entry=0x624e70, retrn=retrn
entry=false) at /build/root-6.26.10/core/rint/src/TRint.cxx:501
#29 0x00000000004011bf in main (argc=<optimized out>, argv=0x7ffeb305b028) at /build/root-6.26.10/main/src/rmain.cxx:84
terminate called after throwing an instance of 'cling::InvalidDerefException'
  what():  Trying to dereference null pointer or trying to call routine taking non-null arguments

ShamrockLee · 2023-02-23T19:13:44Z

@veprbl Do you think it's worth it to split the executable wrapping like what I did in the second commit?
If not, I can drop that PR, and access the .debug output manually.

ShamrockLee · 2023-02-23T19:46:45Z

@ofborg build root.unwrapped.debug

ShamrockLee · 2023-02-23T20:32:07Z

@ofborg build root-unwrapped.debug

veprbl · 2023-02-23T20:50:47Z

Building against wrapped root by default defeats the purpose of splitting the debug output. Did you mean to provide an optional wrapper?

ShamrockLee · 2023-02-23T21:21:32Z

@veprbl Yes. It would be something like root-debug aside from 'root'.

BTW, I just observed that separateDebugInfo behaves differently on Darwin -- it doesn't even produce the .debug output like it does on Linux! Not sure how to make a wrapper under such condition.

ShamrockLee · 2023-02-23T21:22:03Z

@ofborg build root.debug

veprbl · 2023-02-23T21:38:27Z

BTW, I just observed that separateDebugInfo behaves differently on Darwin -- it doesn't even produce the .debug output like it does on Linux! Not sure how to make a wrapper under such condition.

Yes, it won't be possible to have that wrapper on macOS, see #203380.

@veprbl Yes. It would be something like root-debug aside from 'root'.

I don't even see a need for the wrapper. You could add a hook in nix-support into root.debug to set NIX_DEBUG_INFO_DIRS.

ShamrockLee · 2023-02-25T13:03:27Z

Alright, I'll just keep it simple and enable the debug build only on Linux.

I don't even see a need for the wrapper. You could add a hook in nix-support into root.debug to set NIX_DEBUG_INFO_DIRS.

How do I add hooks to a specific output?

BTW, loading the debug information causes the stack trace to take ~25s to show on my laptop (when testing with TFile::Open("nonexistence.root")->GetName()). It would be better to keep it optional.

symphorien · 2023-02-25T13:45:03Z

See https://nixos.wiki/wiki/Debug_Symbols for a write up about how to use separate debug info, and https://github.com/edolstra/dwarffs and https://github.com/symphorien/nixseparatedebuginfod for third party projects that make this easier to deal with.

One of the main appeal of separate debug info is that you only download them when you use them, and in the case of root they are sizeable.

veprbl · 2023-02-25T17:42:23Z

How do I add hooks to a specific output?

You'd probably need to hack $debug/nix-support/ yourself. I'd grep nixpkgs for ideas on how it's done.

pkgs/applications/science/misc/root/default.nix

symphorien · 2023-03-05T10:28:57Z

thank you!

ShamrockLee force-pushed the root-debug-info branch from 046dac4 to 81c3724 Compare February 17, 2023 20:41

ShamrockLee changed the title ~~root: -DCMAKE_BUILD_TYPE=RelWithDebInfo~~ root: cmakeBuildType = "RelWithDebInfo" Feb 17, 2023

ShamrockLee marked this pull request as draft February 17, 2023 21:21

ShamrockLee marked this pull request as ready for review February 17, 2023 21:26

ofborg bot requested a review from veprbl February 17, 2023 22:10

ofborg bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Feb 17, 2023

ShamrockLee force-pushed the root-debug-info branch from 81c3724 to 3f12c05 Compare February 19, 2023 19:33

ShamrockLee force-pushed the root-debug-info branch from 3f12c05 to 5ed5749 Compare February 20, 2023 06:54

ShamrockLee marked this pull request as draft February 20, 2023 19:14

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog This PR adds or changes release notes 8.has: documentation This PR adds or changes documentation labels Feb 23, 2023

ShamrockLee marked this pull request as ready for review February 23, 2023 19:11

ShamrockLee force-pushed the root-debug-info branch from 84a43a6 to 5ed5749 Compare February 23, 2023 21:18

github-actions bot removed 8.has: documentation This PR adds or changes documentation 8.has: changelog This PR adds or changes release notes 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS labels Feb 23, 2023

ShamrockLee force-pushed the root-debug-info branch from 5ed5749 to 9060d2d Compare February 25, 2023 12:57

ShamrockLee force-pushed the root-debug-info branch from 9060d2d to 5e9db21 Compare February 27, 2023 07:08

symphorien reviewed Feb 27, 2023

View reviewed changes

pkgs/applications/science/misc/root/default.nix Outdated Show resolved Hide resolved

symphorien reviewed Mar 1, 2023

View reviewed changes

pkgs/applications/science/misc/root/default.nix Outdated Show resolved Hide resolved

root: separateDebugInfo = true on Linux

c89c22a

ShamrockLee force-pushed the root-debug-info branch from 5e9db21 to c89c22a Compare March 4, 2023 07:07

ShamrockLee changed the title ~~root: cmakeBuildType = "RelWithDebInfo"~~ root: separateDebugInfo = true on Linux Mar 4, 2023

symphorien reviewed Mar 4, 2023

View reviewed changes

pkgs/applications/science/misc/root/default.nix Outdated Show resolved Hide resolved

ShamrockLee requested review from symphorien and removed request for veprbl March 4, 2023 13:17

root: specify separateDebugInfo = true for all pltforms

b962265

ShamrockLee changed the title ~~root: separateDebugInfo = true on Linux~~ root: separateDebugInfo = true Mar 4, 2023

ofborg bot requested a review from veprbl March 4, 2023 19:43

veprbl approved these changes Mar 5, 2023

View reviewed changes

figsoda added 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Mar 5, 2023

symphorien merged commit 768a866 into NixOS:master Mar 5, 2023

Uh oh!

Conversation

ShamrockLee commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

veprbl commented Feb 17, 2023

Uh oh!

ShamrockLee commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Feb 17, 2023

Uh oh!

ShamrockLee commented Feb 19, 2023

Uh oh!

veprbl commented Feb 19, 2023

Uh oh!

ShamrockLee commented Feb 20, 2023

Uh oh!

ShamrockLee commented Feb 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

veprbl commented Feb 20, 2023

Uh oh!

ShamrockLee commented Feb 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

veprbl commented Feb 22, 2023

Uh oh!

ShamrockLee commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Feb 23, 2023

Uh oh!

ShamrockLee commented Feb 23, 2023

Uh oh!

ShamrockLee commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

veprbl commented Feb 23, 2023

Uh oh!

ShamrockLee commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Feb 23, 2023

Uh oh!

veprbl commented Feb 23, 2023

Uh oh!

ShamrockLee commented Feb 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

symphorien commented Feb 25, 2023

Uh oh!

veprbl commented Feb 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

symphorien commented Mar 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ShamrockLee commented Feb 17, 2023 •

edited

Loading

ShamrockLee commented Feb 17, 2023 •

edited

Loading

ShamrockLee commented Feb 20, 2023 •

edited

Loading

ShamrockLee commented Feb 20, 2023 •

edited

Loading

ShamrockLee commented Feb 22, 2023 •

edited

Loading

ShamrockLee commented Feb 23, 2023 •

edited

Loading

ShamrockLee commented Feb 23, 2023 •

edited

Loading

ShamrockLee commented Feb 23, 2023 •

edited

Loading

ShamrockLee commented Feb 25, 2023 •

edited

Loading