Building with Eclipse OpenJ9 will soon need libdwarf-dev pkg (AO8/J9 s390x break)

[mstoodle]

Just a heads up that Eclipse OpenJ9 is adding some valuable diagnostic tooling support (called Direct Dump Reader) to our default builds, and that carries a requirement for the libdwarf-dev package to be installed on machines used to build OpenJDK with Eclipse OpenJ9.

Hopefully, it's only a small imposition to additionally install this package on the build machines in your farm. I'll gladly help if desired.

For a bit more info, I've quoted the motivation from our umbrella Issue 378 for your convenience:

The OMR project has been implementing Direct Dump Reader (DDR) support on top of libdwarf.

OpenJ9 needs to enable DDR support and ensure that the functionality in the OMR project is sufficient to provide the rich debug & introspection capabilities J9 has historically supported.

OpenJ9 issues are difficult to debug without this support.

...though I would have said "OpenJ9 issues are more difficult to debug without this support" :) . The DDR based tooling makes it much easier to find and walk through OpenJ9 internal data structures to debug a core file. We have found it to be an invaluable tool for analyzing problems in the JVM itself.

For more background, please see eclipse-openj9/openj9#378 and linked issues.

[gdams]

@mstoodle do you want to put together a pull request to add this to the ansible scripts? https://github.com/AdoptOpenJDK/openjdk-infrastructure/tree/master/ansible/playbooks

[mstoodle]

@gdams Sure, I'll take a crack at it. Given current course and speed, it will probably not materialize until later in the week.

[gdams]

thanks @mstoodle It would be good to get it on the machines before our builds break!

[mstoodle]

OK, um, slightly embarrassed here but, as it turns out, the Ubuntu playbooks, which are used to build OpenJ9 I believe all have libdwarf-dev in the build requirements already. So this becomes a really easy issue to address . I also see libdwarf-devel being installed as an additional build tool in the rhel.yml file for RHEL7 specifically.

Anyway, this dependency is important if we expand the set of platforms on which we build OpenJ9, but for now it would seem we're already covered where we need to be. I suggest we close, but I'll leave it up to @gdams @karianna for the final determination (maybe you're building OpenJ9 on a platform I'm not aware of?)

[karianna]

https://ci.adoptopenjdk.net should list our existing infra hosts and what branches they build. Looks like we're trying to build on:

openjdk9_openj9_build_arm64_linux
openjdk9_openj9_build_ppc64_aix
openjdk9_openj9_build_ppc64le_linux
openjdk9_openj9_build_s390x_linux
openjdk9_openj9_build_x86-64_linux
openjdk9_openj9_build_x86-64_macos
openjdk9_openj9_build_x86-64_windows

[sxa]

@mstoodle is this additional dependency included in the 0.8.0 release of Eclipse OpenJ9? If so, this can presumably be closed off now.

[mstoodle]

It is. I checked the current builds with OpenJ9 and none of them are failing for lack of this dependency, so I think this can be closed.

New builds and new platforms will need to account for the dependency, but that's just BAU.

I'll close.

[smlambert]

@mstoodle @sxa, do either of you know, if this DDR compilation failure on s390x related to not having this issue (need libdwarf dependency)? Seems like it is... failing since Mar 23rd. Note, we should fix the job, so that it actual reports that it failed. Right now, even though the compilation failed, it says Built the SDK!

Excerpt from (https://ci.adoptopenjdk.net/job/openjdk8_openj9_build_s390x_linux/107/console):

a - ../gc/verbose/handler_standard/VerboseHandlerOutputStandard.o
../../../omrmakefiles/rules.mk:380: recipe for target 'dwarf/DwarfScanner.o' failed
Makefile:36: recipe for target 'ddr-scanner' failed
a - MacroInfo.o
a - MacroTool.o
a - java/genBinaryBlob.o
a - java/genBlobJava.o
a - java/genSuperset.o
a - ClassUDT.o
a - ClassType.o
a - Field.o
a - Macro.o
a - Members.o
a - Modifiers.o
a - NamespaceUDT.o
a - EnumMember.o
a - Symbol_IR.o
a - EnumUDT.o
a - Type.o
a - TypedefUDT.o
a - TypePrinter.o
a - UDT.o
a - UnionUDT.o
Makefile:48: recipe for target 'lib' failed
GNUmakefile:266: recipe for target 'ddr' failed
GNUmakefile:32: recipe for target 'phase_omr' failed
OpenJ9.gmk:322: recipe for target 'build-j9' failed
/data/jenkins/workspace/openjdk8_openj9_build_s390x_linux/openjdk/closed/make/Main.gmk:29: recipe for target 'j9vm-build' failed: integer expression expected
Built the JDK!
�

[mstoodle]

I will defer to someone like @keithc-ca on this question, but given this output from configure, it sounds likely @smlambert is correct:

checking dwarf.h usability... no
checking dwarf.h presence... no
checking for dwarf.h... no
checking libdwarf/dwarf.h usability... no
checking libdwarf/dwarf.h presence... no
checking for libdwarf/dwarf.h... no
checking libdwarf.h usability... no
checking libdwarf.h presence... no
checking for libdwarf.h... no
checking libdwarf/libdwarf.h usability... no
checking libdwarf/libdwarf.h presence... no
checking for libdwarf/libdwarf.h... no
checking for library containing dwarf_init... no
checking for library containing dwarf_init... (cached) no
checking for library containing elf_begin... no

Can someone check the configuration of that machine to confirm it does (not?) have libdwarf installed?

[keithc-ca]

Looking a little further back, you'll see:

In file included from dwarf/DwarfScanner.cpp:22:0:
../../../ddr/include/ddr/scanner/dwarf/DwarfScanner.hpp:39:2: error: #error "Need dwarf.h or libdwarf/dwarf.h"
 #error "Need dwarf.h or libdwarf/dwarf.h"
  ^
../../../ddr/include/ddr/scanner/dwarf/DwarfScanner.hpp:47:2: error: #error "Need libdwarf.h or libdwarf/libdwarf.h"
 #error "Need libdwarf.h or libdwarf/libdwarf.h"
  ^

So it would appear that machine does not have libdwarf-dev installed.

[sxa]

It's a SLES12 machine so I'm not even sure if that's in the default repos. A quick search of the package repository doesn't find anything matching libdwarf but it'll need to be investigated further.

For my own reference: machine is build-marist-sles12-s390x-1 at 148.100.110.56

[gdams]

we are going to switch to building on rhel7 once the machine comes back so we should be okay

[sxa]

So does RHEL7 definitely have a dwarf dev package available @gdams?

[smlambert]

Do we have a timeframe for when this will be resolved? (when rhel7 machine comes back)

Right now, the fact that this is failing, blocks all nightly builds for openjdk8-openj9, so I have fixes coming in from the openj9 project that I can't verify, because there has been no nightly build since March 23.

[keithc-ca]

On SLES, I think the package we want is libdwarf-devel.

[sxa]

@smlambert I thought this was only affecting the s390x linux builds as opposed to all of the AO8J9 ones or are you seeing issues on other platforms?

[sxa]

@keithc-ca Got a reference or a SLES12 machine that already has it? As I mentionEd before it didn't seem to be in the default repos:

build-marist-sles12-s390x-1:~ # zypper search libdwarf-devel
Refreshing service 'SMT-http_lxslsmt'.
Loading repository data...
Reading installed packages...
No matching items found.

[smlambert]

No, there are no other build issues on other platforms @sxa555

What I was describing is the fact that if one platform fails to build, no SDKs for any platform get published (see openjdk-build pipeline scripts). So its all or nothing, in terms of what is published and available for download from the api and website (which is ultimately how it should work), which means we should endeavour not to go days where we are broken (or rethink that deployment story).

[smlambert]

related: #279 (comment)

[keithc-ca]

Perhaps this helps? https://www.suse.com/LinuxPackages/packageRouter.jsp?product=server&version=12&service_pack=&architecture=s390x&package_name=libdw1

[sxa]

@keithc-ca That's already on the machine but it only contains the runtime library, not the headers which is what the build is looking for.

[keithc-ca]

Perhaps you can use one of the RPMs available via [1]?

[1] https://rpmfind.net/linux/rpm2html/search.php?query=libdwarf-devel&submit=Search+...&system=&arch=s390x

[sxa]

Yeah always possible to pull random ones from places but would prefer to use something vaguely official (or perhaps grabbing the source from an official repo and utilising that would be the right answer)