[CI] Cannot start test cluster for :x-pack:qa:kerberos-tests in 6.7

[droberts195]

This problem has occurred in more than one build, for example:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+intake/474/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+release-tests/219/console

The console log shows this error:

> Failed to locate seed node transport file [/var/lib/jenkins/workspace/elastic+elasticsearch+6.7+release-tests/x-pack/qa/kerberos-tests/build/cluster/integTestCluster node0/elasticsearch-6.7.1/logs/transport.ports]: timed out waiting for it to be created after 40 seconds

This can be reproduced on the 6.7. branch using:

./gradlew :x-pack:qa:kerberos-tests:integTest

The log of the test node shows this:

java.lang.IllegalStateException: security initialization failed
    at org.elasticsearch.xpack.security.Security.createComponents(Security.java:416) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$11(Node.java:472) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:475) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: configured service key tab file [/home/dave/elasticsearch/x-pack/qa/kerberos-tests/build/cluster/integTestCluster node0/elasticsearch-6.7.1-SNAPSHOT/config/es.keytab] does not exist
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosRealm.<init>(KerberosRealm.java:100) ~[?:?]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosRealm.<init>(KerberosRealm.java:75) ~[?:?]
    at org.elasticsearch.xpack.security.authc.InternalRealms.lambda$getFactories$6(InternalRealms.java:107) ~[?:?]
    at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:192) ~[?:?]
    at org.elasticsearch.xpack.security.authc.Realms.<init>(Realms.java:69) ~[?:?]
    at org.elasticsearch.xpack.security.Security.createComponents(Security.java:485) ~[?:?]
    at org.elasticsearch.xpack.security.Security.createComponents(Security.java:414) ~[?:?]
    ... 20 more

That is a fatal exception that causes the node to stop.

So the problem is configured service key tab file ... does not exist.

[elasticmachine]

Pinging @elastic/es-security

[bizybot]

We have done some changes recently to dockerize our Kerberos test fixtures, I will investigate this further as I feel its related to this. Thank you.

[droberts195]

The machine I reproduced the problem on did not have Docker installed. It could just be a case that the error message is unfriendly when Docker is not installed.

[droberts195]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+artifactory/42/console is another case. Just before the failure is this:

Removing network b0f5a62c6b5e4e932b7f156ba449cc7d_krb5kdc-fixture__default
Network b0f5a62c6b5e4e932b7f156ba449cc7d_krb5kdc-fixture__default not found.

[droberts195]

This suite is continuing to fail regularly on the 6.7 branch.

I disabled the integTest task for this suite on the 6.7 branch (only) in 34b5ace.

[bizybot]

I see, the krb5kdc-fixture has been disabled, #40624, will pick it up and fix this. Thank you.