change java path search + java 10 support

[pszynk]

Hi

I think that it's a bad idea to use locate in do.py to search for jni.h header file

• it requires locate
• can find the wrong file (happened to me)

If we search for a java binary we can find jre or jdk directory and then get to the include folder.

Also in java 10 there is no javah binary, only javac with flag -h.

[nikohansen]

To me it looks good. I can't however speak so much to the Java specifics. Anybody else can confirm, @brockho @asmaatamna ?

[pszynk]

Btw I wouldn't call this the perfect solution, I'm not sure if it addresses all Linux distributions. Still, I think it's better to look for the Java executable then file named jni.h with locate which is not installed on some distros by default. The most reliable way is to allow the user to pass valid java path by JAVA_HOME or JDK_HOME env, so I've added this option.

[nikohansen]

@brockho @asmaatamna , could you have a look at it and in case merge?

[asmaatamna]

The code looks fine for me as well. I don't know much about Java files location under Linux though. I checked quickly on my Mac and the path returned after jdkpath1 = join(jdkhome, 'include') (line 698 in do.py) does not seem to exist under Mac Os, but I don't think it's supposed to.

[brockho]

Testing the proposed code on our Jenkins linux slave revealed that with the new code exactly the jni.h header was not found anymore:

Started by upstream project "Coco-test2-linux" build number 417
originally caused by:
 Started by GitHub push by brockho
Building remotely on numbbo-ubuntu-12p04-i386 in workspace /builds/disk/builds/workspace/Coco-test2-linux/dotype/build-java/label/numbbo-ubuntu-12p04-i386/pyversion/python
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/numbbo/coco/ # timeout=10
Fetching upstream changes from https://github.com/numbbo/coco/
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress https://github.com/numbbo/coco/ +refs/heads/*:refs/remotes/origin/*
Checking out Revision 7e83eb4871539a3dda987ac9e68b7bc4d5ef9196 (refs/remotes/origin/devel-test2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7e83eb4871539a3dda987ac9e68b7bc4d5ef9196
Commit message: "Merge branch 'development' of https://github.com/numbbo/coco into dimo-dev-test-pr1741"
 > git rev-list bd327e6a5b155ecb1cc5cd47f3b7044e3bb9d2fb # timeout=10
[python] $ /bin/sh -xe /tmp/jenkins9070551172110650703.sh
+ python do.py build-java
AML	['code-experiments/src/coco_random.c', 'code-experiments/src/coco_suite.c', 'code-experiments/src/coco_observer.c', 'code-experiments/src/coco_archive.c', 'code-experiments/src/coco_runtime_c.c'] -> code-experiments/build/java/coco.c
EXPAND	code-experiments/build/java/coco.c.in to code-experiments/build/java/coco.c
EXPAND	code-experiments/src/coco.h to code-experiments/build/java/coco.h
WRITE	code-experiments/build/java/REVISION
WRITE	code-experiments/build/java/VERSION
RUN	javac CocoJNI.java in code-experiments/build/java
RUN	javah CocoJNI in code-experiments/build/java
RUN	gcc -I C:\Program Files\Java\jdk1.8.0_66/include -I C:\Program Files\Java\jdk1.8.0_66/include/linux -c CocoJNI.c in code-experiments/build/java
ERROR: return value=1
CocoJNI.c:12:17: fatal error: jni.h: No such file or directory
compilation terminated.

Traceback (most recent call last):
  File "do.py", line 1039, in <module>
    main(sys.argv[1:])
  File "do.py", line 1003, in main
    elif cmd == 'build-java': build_java()
  File "do.py", line 702, in build_java
    verbose=_verbosity)
  File "/builds/disk/builds/workspace/Coco-test2-linux/dotype/build-java/label/numbbo-ubuntu-12p04-i386/pyversion/python/code-experiments/tools/cocoutils.py", line 121, in run
    universal_newlines=True)
  File "/builds/disk/builds/workspace/Coco-test2-linux/dotype/build-java/label/numbbo-ubuntu-12p04-i386/pyversion/python/code-experiments/tools/cocoutils.py", line 43, in check_output_with_print
    output = check_output(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['gcc', '-I', 'C:\\Program Files\\Java\\jdk1.8.0_66/include', '-I', 'C:\\Program Files\\Java\\jdk1.8.0_66/include/linux', '-c', 'CocoJNI.c']' returned non-zero exit status 1
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user sed s/x/pszynk/  [email protected]
An attempt to send an e-mail to empty list of recipients, ignored.
[Set GitHub commit status (universal)] ERROR on repos [GHRepository@13ebb8f1[description=Numerical Black-Box Optimization Benchmarking Framework,homepage=http://coco.gforge.inria.fr/,name=coco,fork=false,size=211768,milestones={},language=Python,commits={},source=<null>,parent=<null>,responseHeaderFields={null=[HTTP/1.1 200 OK], Access-Control-Allow-Origin=[*], Access-Control-Expose-Headers=[ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval], Cache-Control=[private, max-age=60, s-maxage=60], Content-Encoding=[gzip], Content-Security-Policy=[default-src 'none'], Content-Type=[application/json; charset=utf-8], Date=[Wed, 09 May 2018 11:56:35 GMT], ETag=[W/"c4539c5db471923a68583a6db8b14f71"], Last-Modified=[Tue, 08 May 2018 11:51:12 GMT], OkHttp-Received-Millis=[1525866995482], OkHttp-Response-Source=[NETWORK 200], OkHttp-Selected-Protocol=[http/1.1], OkHttp-Sent-Millis=[1525866995317], Referrer-Policy=[origin-when-cross-origin, strict-origin-when-cross-origin], Server=[GitHub.com], Status=[200 OK], Strict-Transport-Security=[max-age=31536000; includeSubdomains; preload], Transfer-Encoding=[chunked], Vary=[Accept, Authorization, Cookie, X-GitHub-OTP], X-Accepted-OAuth-Scopes=[repo], X-Content-Type-Options=[nosniff], X-Frame-Options=[deny], X-GitHub-Media-Type=[github.v3; format=json], X-GitHub-Request-Id=[CC5C:4F41:F0429E:204BC26:5AF2E1EF], X-OAuth-Scopes=[admin:repo_hook, gist, repo, user], X-RateLimit-Limit=[5000], X-RateLimit-Remaining=[4781], X-RateLimit-Reset=[1525868412], X-Runtime-rack=[0.075240], X-XSS-Protection=[1; mode=block]},url=https://api.github.com/repos/numbbo/coco,id=40402006]] (sha:7e83eb4) with context:Coco-test2-linux/dotype=build-java,label=numbbo-ubuntu-12p04-i386,pyversion=python
Setting commit status on GitHub for https://github.com/numbbo/coco/commit/7e83eb4871539a3dda987ac9e68b7bc4d5ef9196
Finished: FAILURE

We should therefore not accept the pull request at the moment.

[pszynk]

Jenkins runs this test on the Linux machine right? Path to jni.h starts with C:\Program Files\Java\jdk1.8.0_66 which is rather strange. Are you sure that Jenkins isn't setting JAVA_HOME or JDK_HOME environment variable for the Linux machine like it was a Windows one? JAVA_HOME and JDK_HOME are prioritized, which I guess should be mentioned in documentation.

[brockho]

Good catch for the Windows/Linux issue. I don't know at all what is going on. A bit more investigation gave that

jdkhome = os.environ.get('JAVA_HOME') or os.environ.get('JDK_HOME')

gives indeed C:\Program Files\Java\jdk1.8.0_66 during the tests although the same line, when run on the exact same machine in an ipython shell by hand, gives correctly /usr/lib/jvm/java-8-oracle/.

Moreover, calling

 grep -r JAVA_HOME ~/

from the shell (as user ci) does not give any other occurrence of JAVA_HOME than the ones in our do.py.

Don't know whether this helps.

[brockho]

@nikohansen suggested today to use a system call (via check_output in code-experiments/tools/cocoutils.py) to get the path to the Java home under Linux instead of the os.environ.

[brockho]

According to https://stackoverflow.com/questions/1117398/java-home-directory-in-linux (thanks @ttusar ), for linux and mac, this gets us the java_home:

java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home'

For windows, it's slightly different:

java -XshowSettings:properties -version 2>&1 | findstr "java.home"

[ttusar]

BTW, this last trick can only be used since JDK1.7

[brockho]

Just FTR: the final "trick" was to simply abandon the if clause that caused all the previous problems. CircleCI and Jenkins ran through all the tests successfully.