system tests failing on aix due to libcrypto could not be loaded

[pshipton]

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-aix_ppc-64_cmprssptrs/107/console
https://ci.eclipse.org/openj9/job/Test-extended.system-JDK8-aix_ppc-64_cmprssptrs/58/console

03:47:33 clone_stf:
03:47:33      [exec] Cloning into 'stf'...
03:47:33      [exec] exec(): 0509-036 Cannot load program /opt/freeware/libexec/git-core/git-remote-https because of the following errors:
03:47:33      [exec] 	0509-022 Cannot load module /usr/lib/libcurl.a(libcurl.so.4).
03:47:33      [exec] 	0509-150   Dependent module /home/u0020236/workspace/Test-sanity.system-JDK8-aix_ppc-64_cmprssptrs/openjdkbinary/j2sdk-image/jre/lib/ppc64/libcrypto.a(libcrypto.so) could not be loaded.
03:47:33      [exec] 	0509-152   Member libcrypto.so is not found in archive 
03:47:33      [exec] 	0509-022 Cannot load module git-remote-https.
03:47:33      [exec] 	0509-150   Dependent module /usr/lib/libcurl.a(libcurl.so.4) could not be loaded.
03:47:33      [exec] 	0509-022 Cannot load module .
03:47:33      [exec] Result: 128

[pshipton]

@mbvreddy

[pshipton]

@mbvreddy @enasser any ideas about this one?

[mbvreddy]

It is trying to load libcurl.a and its dependent library libcrypto.so (in libcrypto.a). libcrypto.a from OpenSSL 1.1.1 bundled with JDK has only libcrypto.so.1.1. It doesn't have libcrypto.so. Hence, it failed. I am wondering why libcrypto.a from JDK directory is loaded. It should load libcrypto.a that contains required version of libcrypto.so

[pshipton]

@Mesbah-Alam @smlambert can you please help with this one. I suspect the tests are setting LIBPATH or similar to contain Test-sanity.system-JDK8-aix_ppc-64_cmprssptrs/openjdkbinary/j2sdk-image/jre/lib/ppc64 before the system locations, and this is causing the problem.

[Mesbah-Alam]

openj9Settings.mk does set LIBPATH here: https://github.com/eclipse/openj9/blob/b03cae14f7322dc04e72874128adc8150d91028f/test/TestConfig/openj9Settings.mk#L80.

Looking at the code in the above mk file, one of the values being set in LIBPATH would be "$(JAVA_LIB_DIR)$(D)$(ARCH_DIR)$(D)$(VM_SUBDIR)", which, for this test would be: Test-sanity.system-JDK8-aix_ppc-64_cmprssptrs/openjdkbinary/j2sdk-image/jre/lib/ppc64/compressedrefs

---

@pshipton - this code has been there for a while. Is this a new problem on AIX? Can it be an AIX issue with the Curl installation?

I did a google with the error and found this: nodejs/build#551

There is a comment by @sxa555 here that might be of interest:

It should work if LIBPATH is pointing at /usr/lib (although that could well cause other issues of course!) - the libcrypto.a in /opt/freeware/lib doesn't have a non-version-suffixed version of libcrypto.so in it but the one in /usr/lib does. I'm not sure why the one in /opt/freeware/lib doesn't ... You'd expect it to work with the version of curl in there. You could just override it for the execution of curl for now.

[pshipton]

@Mesbah-Alam this is a new problem as libcrypto was recently added to the VM directory.

• do we still need to add the VM directory to the LIBPATH?
• if so can we add it to the end rather than the beginning? Not sure if this will resolve the problem but its worth a shot.

[Mesbah-Alam]

@pshipton -

do we still need to add the VM directory to the LIBPATH?

• I spoke to Sophia Guo, and she confirmed that lots of functional tests use this value in LIBPATH. So, we do need it.

if so can we add it to the end rather than the beginning? Not sure if this will resolve the problem but its worth a shot.

• I have tried it, and it did not work, we still have the same error : https://hyc-runtimes-jenkins.swg-devops.com/view/Test_system/job/Grinder/495/console

[pshipton]

@Mesbah-Alam Weird, because when I look at a machine the LIBPATH is set to /usr/lib and this directory does contain libcrypto.a. Are you sure the changes took effect in the grinder? I don't see any confirmation in the output.

[Mesbah-Alam]

The changes should have taken effect, from the job output:

12:51:57 get testKitGen and functional test material...
12:51:57 git clone -b openj9-openjdk-jdk8-issues-129-fliplibpathorder https://github.com/Mesbah-Alam/openj9.git

So it's checking out openj9 from the branch where the change was made..

[Mesbah-Alam]

Logging into the machine pll011.rtp.raleigh.ibm.com where this Grinder ran , I also see the change present in /home/j9build/workspace/Grinder/openjdk-tests/TestConfig/openj9Settings.mk :

 +79          endif
   +80          ADD_JVM_LIB_DIR_TO_LIBPATH:=export LIBPATH=$(Q)$(LIBPATH)$(P)$(JAVA_LIB_DIR)$(D)$(VM_SUBDIR)$(P)$(JAVA_SHARED_LIBRARIES_DIR)$(P)$(JAVA_BIN)$(D)j9vm$(Q);
   +81  else

[pshipton]

@keithc-ca @mikezhang1234567890 any ideas?

[Mesbah-Alam]

Logging into the machine and issuing the make commands locally to compile the test makes it work. It could be a grinder issue which is causing the change to not get picked up! I've created a PR with the change https://github.com/eclipse/openj9/pull/3289/files

[keithc-ca]

Some environment variables get set as part of an interactive login; perhaps the jenkins slave doesn't have LIBPATH set as expected.

[pshipton]

I think we are good, Mesbah's fix is working. The initial tests were not running with the fix.

[pshipton]

Fixed by eclipse-openj9/openj9#3289

[pshipton]

Still failing, I've put comments into eclipse-openj9/openj9#3289

[Mesbah-Alam]

Still failing : https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-aix_ppc-64_cmprssptrs/112/console

[pshipton]

@Mesbah-Alam can you please scour the code again for any places we might have missed.

[Mesbah-Alam]

I did another search. I can not find any other place where LIBPATH is being exported other than the places we have already changed. The problem seems to be somewhere else.

[pshipton]

I believe its the VM itself which is setting the LIBPATH and causing this problem. The VM sets the LIBPATH as follows, and I expect anything exec'ed from the VM will inherit this.

2CIENVVAR LIBPATH=/bluebird/builds/bld_399771/sdk/ap6480/jre/lib/ppc64:/bluebird/builds/bld_399771/sdk/ap6480/jre/lib/ppc64/ compressedrefs:/bluebird/builds/bld_399771/sdk/ap6480/jre/lib/ppc64/j9vm:/bluebird/builds/bld_399771/sdk/ap6480/jre/lib/ppc64:/bl uebird/builds/bld_399771/sdk/ap6480/jre/../lib/ppc64:/bluebird/builds/bld_399771/sdk/ap6480/jre/lib/icc:/usr/lib:/usr/lib

[pshipton]

Disabling openssl on AIX at OpenJ9 eclipse-openj9/openj9#3334

[pshipton]

Created an issue at Adopt as well adoptium/temurin-build#658

[sxa]

I've hit issues before with the AIX JVM replacing things in the LIBPATH and causing problems. Splatting LIBPATH before the operations spawned from java would work too. Disabling it seems a bit drastic though - could we not just stop it bundling, or even just require the user to manually adjust the LIBPATH if they want to use it?

[pshipton]

@sxa555 We could just stop bundling. This will cause the VM to go looking for a libcrypto.a I believe. If everything works correctly, it would typically find the wrong version on the system, fail to load the JNI library, and fail back to Java crypto code. There is some risk of it not working correctly and then I'm not sure what will happen, although @mbvreddy tells me it works. If you want to try not bundling at Adopt we can see if any problems occur.

For OpenJ9 since we don't have the correct libcrypto library on the machines, disabling the openssl support seemed the safer short term alternative to ensure that AIX testing would proceed without problems for the 0.11.0 release.

If the JVM is created with openssl support enabled, a user (including OpenJ9) could build an openssl 1.1 library and make it available on the machine for the VM to find. However this may just result in similar problems as described in this Issue, depending on how the user makes the crypto library available to the VM. I expect there are workarounds, which require experimentation and documentation. This can be sorted out in the next week or so I expect.

[groeges]

OpenSSL should now be working on all platforms. Closing issue.