Hadoop 2.7.x is not compatible with Java 9

[jaymode]

In the CI tests for Java 9, we are getting failures in the hdfsFixture:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
    Exception in thread "main" java.lang.ExceptionInInitializerError
    	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
    	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
    	at hdfs.MiniHDFS.main(MiniHDFS.java:89)
    Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
    	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116)
    	at java.base/java.lang.String.substring(String.java:1885)
    	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51)
    	... 5 more

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/3051/console

This is a known issue with Hadoop 2.7.x, https://issues.apache.org/jira/browse/HADOOP-14586, and it may not be fixed. A "hacky" workaround was used by the reported of the issue, https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=commitdiff;h=85a27a2. Maybe we can bring our version of the client jars up to a newer version?

[martijnvg]

Today's failures:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/3059/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.5+java9-periodic/266/
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/3058
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/3058/
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/3055
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.5+java9-periodic/265
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+java9-periodic/3037/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.5+java9-periodic/264/console

[martijnvg]

Similar failure in smoke test plugins qa module:

plugins/build/cluster/integTestCluster node0/elasticsearch-5.5.0-SNAPSHOT]
|    [2017-06-29T00:49:49,300][WARN ][o.e.n.Node               ] version [5.5.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
|    [2017-06-29T00:49:52,621][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [main], exiting
:x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#wait FAILED
:x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#wait (Thread[Daemon worker,5,main]) completed. Took 4.544 secs.
|    java.lang.ExceptionInInitializerError: null
|    	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) ~[?:?]
|    	at java.lang.Class.forName0(Native Method) ~[?:?]
:x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#stop (Thread[Daemon worker,5,main]) started.
|    	at java.lang.Class.forName(Class.java:292) ~[?:?]
|    	at org.elasticsearch.repositories.hdfs.HdfsPlugin.evilHadoopInit(HdfsPlugin.java:65) ~[?:?]
|    	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
|    	at org.elasticsearch.repositories.hdfs.HdfsPlugin.<clinit>(HdfsPlugin.java:48) ~[?:?]
|    	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
|    	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
|    	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
|    	at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?]
|    	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:426) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:140) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.node.Node.<init>(Node.java:312) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
:x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#stop
Putting task artifact state for task ':x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#stop' into context took 0.0 secs.
Executing task ':x-pack-elasticsearch:qa:smoke-test-plugins:integTestCluster#stop' (up-to-date check took 0.0 secs) due to:
|    	at org.elasticsearch.node.Node.<init>(Node.java:244) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
  Task has not declared any outputs.
|    	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
Shutting down external node with pid 13495
|    	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
Starting process 'command 'kill''. Working directory: /var/lib/jenkins/workspace/elastic+x-pack-elasticsearch+5.5+java9-periodic/elasticsearch-extra/x-pack-elasticsearch/qa/smoke-test-plugins Command: kill -9 13495
|    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
Successfully started process 'command 'kill''
|    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.5.0-SNAPSHOT.jar:5.5.0-SNAPSHOT]
|    Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
|    	at java.lang.String.checkBoundsBeginEnd(String.java:3116) ~[?:?]
|    	at java.lang.String.substring(String.java:1885) ~[?:?]
|    	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51) ~[?:?]
|    	... 25 more

Build url: https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+5.5+java9-periodic/90/console

another instance: https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+5.x+java9-periodic/475/console

[jbaiera]

We can look into bringing the Hadoop version up to 2.8.x to fix this, as I believe that this problem is fixed in that release line. It's frustrating that 2.7 may simply not allow java 9 compatibility at all. I'm not a fan of the hack approach since it may have unintended side effects down the road on code that is prepared for the version change.

[jbaiera]

I've got the plugin upgraded locally, but I'm running into a problem with 2.8.0 in that there is a thread started in the HDFS client that leaks and completely ignores thread interrupts. Seems like it's been reported but the fix isn't targeted for release until 2.9.0: https://issues.apache.org/jira/browse/HADOOP-12829

Edit: Got a work around figured out for our testing environment. PR will land soon.

[droberts195]

There have been 23 failures so far today. If the PR cannot be merged today then I think it would be worth disabling the test to avoid the noise over the weekend.

[droberts195]

Also another 10 cases of the smoke test failure detailed in #25450 (comment)