[SR-2136] LLDB asserting on missing glibc module: Ubuntu, master-next, build layout only

[trfiala]

Previous ID	SR-2136
Radar	rdar://problem/27459201
Original Reporter	@trfiala
Type	Bug
Status	Closed
Resolution	Done

Environment

swift-lldb/master-next branch, Ubuntu 16.04 x86_64, Ubuntu 14.04 x86_64. This happens only with the build layout of lldb/swift. It does not manifest when using the install layout of lldb and swift.

Additional Detail from JIRA

Votes	0
Component/s	LLDB for Swift
Labels	Bug, Linux
Assignee	@trfiala
Priority	Medium

md5: 6e445793b80bac593e7606ca7900ace4

relates to:

• SR-1300 LLDB REPL tests fail on Ubuntu when not run from install layout

Issue Description:

After merging LLVM.org LLDB trunk @r276079 into swift-lldb/master-next, we're getting an assert in many of the LLDB Swift tests here:

swift/lib/ClangImporter/ClangImporter.cpp:402: Assertion !GlibcModuleMapPath.empty() failed

It looks like we're not finding a resource dir to hand over to Swift when we create a clang importer as part of setting up a SwiftASTContext for each module. This might be a temporal issue where we would have found it later.

[trfiala]

I'm going to grab this since the most likely candidate for addressing is out this week.

[trfiala]

The backtrace for the assert looks like this when running one of the Python-driven LLDB tests against a Swift test subject:

(lldb) thread list
Process 18364 stopped

• thread [DON'T MERGE] Test pull request #1: tid = 18364, 0x00007f29bbb06267 libc.so.6`__GI_raise(sig=6) + 55 at raise.c:55, name = 'python', stop reason = signal SIGABRT
  (lldb) bt
• thread [DON'T MERGE] Test pull request #1: tid = 18364, 0x00007f29bbb06267 libc.so.6`__GI_raise(sig=6) + 55 at raise.c:55, name = 'python', stop reason = signal SIGABRT
• frame #0: 0x00007f29bbb06267 libc.so.6`__GI_raise(sig=6) + 55 at raise.c:55
  frame [DON'T MERGE] Test pull request #1: 0x00007f29bbb07eca libc.so.6`__GI_abort + 362 at abort.c:89
  frame [lldb][NFC] Actually test which method we call in TestCallOverriddenMethod #2: 0x00007f29bbaff03d libc.so.6`__assert_fail_base(fmt=<unavailable>, assertion=<unavailable>, file=<unavailable>, line=<unavailable>, function=<unavailable>) + 285 at assert.c:92
  frame [lldb] Fix merge after r375160 #3: 0x00007f29bbaff0f2 libc.so.6`GI_assert_fail(assertion=<unavailable>, file=<unavailable>, line=<unavailable>, function=<unavailable>) + 66 at assert.c:101
  frame Apple/stable/20190619 #4: 0x00007f29b39a234c _lldb.so`getNormalInvocationArguments(invocationArgStrs=size=512, ctx=0x000000000122b270, importerOpts=0x000000000122ac68) + 3292 at ClangImporter.cpp:402
  frame [compiler-rt] Use signal on Windows #5: 0x00007f29b39a0352 _lldb.so`swift::ClangImporter::create(ctx=0x000000000122b270, importerOpts=0x000000000122ac68, tracker=0x0000000000000000) + 290 at ClangImporter.cpp:570
  frame Cherry-pick "Fix darwin-ld-lto test for some special path" to stable #6: 0x00007f29b300c10e _lldb.so`lldb_private::SwiftASTContext::GetClangImporter(this=0x000000000122a5a0) + 222 at SwiftASTContext.cpp:3299
  frame Test pull request system #7: 0x00007f29b300d9ec _lldb.so`lldb_private::SwiftASTContext::CreateInstance(language=eLanguageTypeSwift, module=0x00000000011dd170, target=0x0000000000000000, extra_options=0x0000000000000000) + 5404 at SwiftASTContext.cpp:1730
  frame Recover from a bad merge #8: 0x00007f29b309a3de _lldb.so`lldb_private::TypeSystem::CreateInstance(language=eLanguageTypeSwift, module=0x00000000011dd170) + 126 at TypeSystem.cpp:36
  frame [Monorepo fixup] Reset lld contents in the swift/master branch to mat… #9: 0x00007f29b309af36 _lldb.so`lldb_private::TypeSystemMap::GetTypeSystemForLanguage(this=0x00000000011dd2c8, language=eLanguageTypeSwift, module=0x00000000011dd170, can_create=true) + 470 at TypeSystem.cpp:278
  frame Cherrypick upstream reproducer changes. #10: 0x00007f29b2e48fb4 _lldb.so`lldb_private::Module::GetTypeSystemForLanguage(this=0x00000000011dd170, language=eLanguageTypeSwift) + 52 at Module.cpp:430
  frame [VirtualFileSystem] Support virtual working directory in the RedirectingFS #11: 0x00007f29b3237512 _lldb.so`SymbolFileDWARF::GetTypeSystemForLanguage(this=0x00000000012281d0, language=eLanguageTypeSwift) + 130 at SymbolFileDWARF.cpp:519
  frame Fix compiler errors. #12: 0x00007f29b3239101 _lldb.so`SymbolFileDWARF::ParseCompileUnitFunction(this=0x00000000012281d0, sc=0x00007ffe6e2559b0, die=0x00007ffe6e2554f8) + 97 at SymbolFileDWARF.cpp:1008
  frame [CMake] Fix the static bindings workflow. #13: 0x00007f29b323da14 _lldb.so`SymbolFileDWARF::ResolveSymbolContext(this=0x00000000012281d0, so_addr=0x00007ffe6e255938, resolve_scope=127, sc=0x00007ffe6e2559b0) + 804 at SymbolFileDWARF.cpp:2096
  frame Upstream arm64e and Pointer Authentication support #14: 0x00007f29b3074950 _lldb.so`lldb_private::SymbolVendor::ResolveSymbolContext(this=0x00000000012255a0, so_addr=0x00007ffe6e255938, resolve_scope=127, sc=0x00007ffe6e2559b0) + 192 at SymbolVendor.cpp:302
  frame [LLDB] Cherrypick tablegen dependencies #15: 0x00007f29b2e49878 _lldb.so`lldb_private::Module::ResolveSymbolContextForAddress(this=0x00000000011dd170, so_addr=0x00007ffe6e255938, resolve_scope=127, sc=0x00007ffe6e2559b0, resolve_tail_call_address=false) + 568 at Module.cpp:567
  frame monorepo fixup: Merge --theirs all the untracked dirs in swift/master #16: 0x00007f29b2dca6f7 _lldb.so`lldb_private::Address::CalculateSymbolContext(this=0x00007ffe6e255938, sc=0x00007ffe6e2559b0, resolve_scope=127) const + 231 at Address.cpp:869
  frame move registrations out of SwiftREPL::CompleteCode #17: 0x00007f29b2fe48e1 _lldb.so`lldb_private::CompileUnit::ResolveSymbolContext(this=0x0000000001228e70, file_spec=0x0000000001225630, line=16, check_inlines=true, exact=false, resolve_scope=127, sc_list=0x00007ffe6e255c60) + 769 at CompileUnit.cpp:425
  frame [SwiftLanguageRuntime] Get rid of some legacy. #18: 0x00007f29b35e336a _lldb.so`lldb_private::BreakpointResolverFileLine::SearchCallback(this=0x0000000001225610, filter=0x0000000001225280, context=0x00007ffe6e255d28, addr=0x0000000000000000, containing=false) + 506 at BreakpointResolverFileLine.cpp:87
  frame [lldb] Adjust for the new class_rw_t layout. #19: 0x00007f29b2e91442 _lldb.so`lldb_private::SearchFilter::DoModuleIteration(this=0x0000000001225280, context=0x00007ffe6e255f00, searcher=0x0000000001225610) + 626 at SearchFilter.cpp:191
  frame Reintroduce some seemingly unused code. A test fails. #20: 0x00007f29b2e911a5 _lldb.so`lldb_private::SearchFilter::Search(this=0x0000000001225280, searcher=0x0000000001225610) + 245 at SearchFilter.cpp:118
  frame Fix a couple of SwiftREPL crashes. #21: 0x00007f29b35e09d1 _lldb.so`lldb_private::BreakpointResolver::ResolveBreakpoint(this=0x0000000001225610, filter=0x0000000001225280) + 33 at BreakpointResolver.cpp:62
  frame [TableGen] Fix default and description of UseAllCompilerFlags #22: 0x00007f29b2d431d3 _lldb.so`lldb_private::Breakpoint::ResolveBreakpoint(this=0x00000000012252a0) + 115 at Breakpoint.cpp:355
  frame Skip TestIndirectEnumVariables.py tests on macOS (56608695) #23: 0x00007f29b315eac5 _lldb.so`lldb_private::Target::AddBreakpoint(this=0x0000000001211630, bp_sp=lldb_private::Breakpoint @ 0x00000000012252a0, internal=false) + 341 at Target.cpp:714
  frame XFAIL TestIndirectEnumVariables.py #24: 0x00007f29b315fe03 _lldb.so`lldb_private::Target::CreateBreakpoint(this=0x0000000001211630, filter_sp=lldb_private::SearchFilter @ 0x0000000001225280, resolver_sp=lldb_private::BreakpointResolver @ 0x0000000001225610, internal=false, request_hardware=false, resolve_indirect_symbols=true) + 307 at Target.cpp:691
  frame Dwarfdump base addr entry support #25: 0x00007f29b31602b8 _lldb.so`lldb_private::Target::CreateBreakpoint(this=0x0000000001211630, containingModules=0x0000000000000000, file=0x0000000001222480, line_no=16, offset=0, check_inlines=eLazyBoolYes, skip_prologue=eLazyBoolYes, internal=false, hardware=false, move_to_nearest_code=eLazyBoolYes) + 1144 at Target.cpp:432
  frame LLD cherry-picks #26: 0x00007f29b19116c9 _lldb.so`lldb::SBTarget::BreakpointCreateByLocation(this=0x0000000001222460, sb_file_spec=0x00007ffe6e257470, line=16, offset=0) + 361 at SBTarget.cpp:826
  frame [Driver] Force llvm to install its handlers before lldb's #27: 0x00007f29b191154d _lldb.so`lldb::SBTarget::BreakpointCreateByLocation(this=0x0000000001222460, sb_file_spec=0x00007ffe6e257470, line=16) + 45 at SBTarget.cpp:805
  frame Revert "XFAIL TestIndirectEnumVariables.py" #28: 0x00007f29b1911503 _lldb.so`lldb::SBTarget::BreakpointCreateByLocation(this=0x0000000001222460, file="main.swift", line=16) + 83 at SBTarget.cpp:798
  frame Reapply: Port swift specific compiler-rt code to Windows #29: 0x00007f29b1a2b8be lldb.so`wrap_SBTarget_BreakpointCreateByLocation_SWIG_0((null)=0x0000000000000000, args=0x00007f29abf42500) + 574 at LLDBWrapPython.cpp:47643
  frame Static analyzer cherrypicks 1 #30: 0x00007f29b19e112b _lldb.so`_wrap_SBTarget_BreakpointCreateByLocation(self=0x0000000000000000, args=0x00007f29abf42500) + 683 at LLDBWrapPython.cpp:47801
  frame ValueObject: Fix a crash related to children address type computation #31: 0x00000000004c0b69 python`PyEval_EvalFrameEx + 28521
  frame [TSan] Fix CI test failures #32: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame More fixup for "[TSan] Turn on ignore_interceptors_accesses" #33: 0x00000000004bfc61 python`PyEval_EvalFrameEx + 24673
  frame Disable the 2 failing tests for now to make the branch baseline green #34: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame [lit] Drop the user-site packages directory from search paths when ru… #35: 0x00000000004bfc61 python`PyEval_EvalFrameEx + 24673
  frame Cherry-pick bugfixes #36: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame [TSan] Fix test failing on Linux #37: 0x00000000004bfc61 python`PyEval_EvalFrameEx + 24673
  frame Cherry-pick fixes for rdar://problem/48167864 #38: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Disable Clang modules on swift-lldb. #39: 0x00000000004bfc61 python`PyEval_EvalFrameEx + 24673
  frame Revert "Disable Clang modules on swift-lldb." #40: 0x00000000004bf10f python`PyEval_EvalFrameEx + 21775
  frame lldb: correct default value for Boolean #41: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Add JSON file for swift-5.1-branch monorepo checkout; skip swift-ast-… #42: 0x00000000004d3cf3 python`__lldb_unnamed_symbol3329$$python + 387
  frame Disable clang modules in LLDB #43: 0x00000000004a45de python`PyObject_Call + 62
  frame Fast forward apple/stable/20190619 to Sep 4. #44: 0x00000000004bc9bc python`PyEval_EvalFrameEx + 11708
  frame Cherry-pick more commits #45: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Avoid adding any framework paths in /System/Library in process_one_mo… #46: 0x00000000004d3bb9 python`___lldb_unnamed_symbol3329$$python + 73
  frame Avoid adding any framework paths in /System/Library in process_one_mo… #47: 0x00000000004ece7e python`___lldb_unnamed_symbol3444$$python + 446
  frame Revert "ValueObject: Fix a crash related to children address type com… #48: 0x00000000004a45de python`PyObject_Call + 62
  frame [lldb][NFC] Make LLVMUserExpression::DoExecute return early #49: 0x0000000000546a93 python`___lldb_unnamed_symbol3973$$python + 51
  frame [test] Rename Modules to ObjectFile (NFC) #50: 0x00000000004bf3cc python`PyEval_EvalFrameEx + 22476
  frame [ClangExpressionDeclMap] Remove divergence from upstream lldb. #51: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Mark all Swift-specific changes in API with comments. NFC #52: 0x00000000004bf4e9 python`PyEval_EvalFrameEx + 22761
  frame Cherrypick/latest reproducers #53: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Mark all Swift-specific changes in Breakpoint with comments. NFC #54: 0x00000000004bf4e9 python`PyEval_EvalFrameEx + 22761
  frame [SystemInitializerCommon] Remove divergence from upstream. #55: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame [MIPS/MIPS64] Remove another set of differences from upstream. #56: 0x00000000004d3cf3 python`___lldb_unnamed_symbol3329$$python + 387
  frame Cherry-pick LiveDebugValues bugfix #57: 0x00000000004a45de python`PyObject_Call + 62
  frame [Diagnostics] Update SwiftASTContext to use the new DiagnosticConsumer interface #58: 0x00000000004bc9bc python`PyEval_EvalFrameEx + 11708
  frame Cleanup: Skip macOS-only test on non-macOS systems and give it a uniq… #59: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame Cleanup: Skip macOS-only test on non-macOS systems and give it a uniq… #60: 0x00000000004d3bb9 python`___lldb_unnamed_symbol3329$$python + 73
  frame Rename a flang test case #61: 0x00000000004ece7e python`___lldb_unnamed_symbol3444$$python + 446
  frame [apple/stable/20190619] Rename a flang test case #62: 0x00000000004a45de python`PyObject_Call + 62
  frame [swift/master] Rename a flang test case #63: 0x0000000000546a93 python`___lldb_unnamed_symbol3973$$python + 51
  frame [TextAPI][TBDGen] Tapi add targets #64: 0x00000000004bf3cc python`PyEval_EvalFrameEx + 22476
  frame test: remove import for lldbsuite.support.fs #65: 0x00000000004bf10f python`PyEval_EvalFrameEx + 21775
  frame [Reproducer] Don't instrument methods that get called from the signal… #66: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame [swift/master-next] [Diagnostics] Update SwiftASTContext to use the new DiagnosticConsumer interface #67: 0x00000000004bfc61 python`PyEval_EvalFrameEx + 24673
  frame debug prints to see what repo/branch bots are checking out #68: 0x00000000004b7986 python`PyEval_EvalCodeEx + 1046
  frame [clang][driver][flang] Mark test as unsupported on darwin #69: 0x00000000004e8f3f python`___lldb_unnamed_symbol3424$$python + 63
  frame Clang-format Commands. (NFC) #70: 0x00000000004e3b02 python`PyRun_FileExFlags + 130
  frame Move cherry-picks from stable/20190619 into swift/master  #71: 0x00000000004e22e6 python`PyRun_SimpleFileExFlags + 390
  frame Also disable flang test on [Fedora] Linux #72: 0x0000000000490fe1 python`Py_Main + 1505
  frame Remove unused parameter in ClangASTContext::CreateInstance #73: 0x00007f29bbaf1a40 libc.so.6`__libc_start_main(main=(python`main), argc=11, argv=0x00007ffe6e259a98, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffe6e259a88) + 240 at libc-start.c:289
  frame [upstreaming] Revert removal of comment in SBAddress.h #74: 0x0000000000490919 python`_start + 41

The issue happens when the test is trying to resolve a breakpoint address by location, and in the process enumerates the swift modules, which creates the SwiftASTContexts, each of which creates a clang module importer, which then gets angry because we haven't told it where to find glibc.

[trfiala]

The LLDB test suite uses LLDB and Swift out of the build dir configuration — i.e. not the final install layout. I'm checking with a full package build install layout to see if the behavior is different. This might end up being the same root cause that led to me turning off the in-tree REPL tests on Linux several weeks ago. If they are the same, it would be good to fix that.

[trfiala]

> The LLDB test suite uses LLDB and Swift out of the build dir configuration — i.e. not the final install layout. I'm checking with a full package build install layout to see if the behavior is different.

That hunch was correct. The LLDB Swift tests all pass when using the install-layout version of lldb. This seems to imply our resource dir lookup logic is not adequate on Linux when attempting to run lldb out of the build dir to either (1) use the repl or (2) debug a Swift program. Our test suite really depends on running in a build-style layout, so this is going to be important to address.

[trfiala]

This issue seems to have the same underlying cause as SR-1300.

[trfiala]

The work-around here is to use an install-layout lldb and not try to use it directly out of the build dir.

Meanwhile, I'm going to see if I can get a real solution to this.

[trfiala]

I managed to make a fix for this. I'm going to run it through a PR after I do a bit more local testing.

[trfiala]

I have a PR testing this fix on swift-lldb/master-next now:
apple/swift-lldb#36

Once I have that tested and submitted successfully, I'll pull this back into swift-lldb/master, and possibly further.

[trfiala]

This is working in the swift-lldb/master-next branch, as witnessed in this Ubuntu 15.10 x86_64 build:
https://ci.swift.org/view/swift-master-next/job/oss-lldb-incremental-linux-ubuntu-15_10-master-next/30/

It was developed on Ubuntu 16.04 x86_64, so works there as well. I'm waiting on the Ubuntu 14.04 x86_64 builder to work through it. Once those are clear, I'll get this in swift-lldb/master.

[trfiala]

This was fixed a while back.