testThatEmailAttachmentsAreSent failed due to internal cluster not forming #30870
Description
This failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/2063/console
The repro command is:
./gradlew :x-pack:plugin:watcher:test -Dtests.seed=81BB68C492DD47D6 -Dtests.class=org.elasticsearch.xpack.watcher.actions.email.EmailAttachmentTests -Dtests.method="testThatEmailAttachmentsAreSent" -Dtests.security.manager=true -Dtests.locale=mk -Dtests.timezone=Etc/UCT
This did not reproduce locally for me.
The cause of the failure is not in the actual test code, but in the formation of the internal test cluster:
> Throwable #1: java.lang.IllegalStateException: cluster failed to form with expected nodes [{node_s2}{FikzomkMTTueLareWsmHlQ}{AjalmjLQQReu3dFLp3FG9w}{127.0.0.1}{127.0.0.1:46204}{xpack.installed=true}, {node_s0}{qExd_lljQ_a26sqD6YRnvQ}{t7MyJIeFT8C7VX9Y3JR3Bw}{127.0.0.1}{127.0.0.1:42826}{xpack.installed=true}, {node_s1}{T-N7bqCeQwyFLg26owJxRw}{g5GAMmHqQHSHWLyjMu_q_g}{127.0.0.1}{127.0.0.1:37292}{xpack.installed=true}] and actual nodes nodes:
> {node_s1}{T-N7bqCeQwyFLg26owJxRw}{g5GAMmHqQHSHWLyjMu_q_g}{127.0.0.1}{127.0.0.1:37292}{xpack.installed=true}, master
> {node_s0}{qExd_lljQ_a26sqD6YRnvQ}{t7MyJIeFT8C7VX9Y3JR3Bw}{127.0.0.1}{127.0.0.1:42826}{xpack.installed=true}
> at __randomizedtesting.SeedInfo.seed([81BB68C492DD47D6:5FF1DE3DEDA219B7]:0)
> at org.elasticsearch.test.InternalTestCluster.validateClusterFormed(InternalTestCluster.java:1084)
> at org.elasticsearch.test.InternalTestCluster.validateClusterFormed(InternalTestCluster.java:1059)
> at org.elasticsearch.test.InternalTestCluster.reset(InternalTestCluster.java:1051)
> at org.elasticsearch.test.InternalTestCluster.beforeTest(InternalTestCluster.java:973)
> at org.elasticsearch.test.ESIntegTestCase.beforeInternal(ESIntegTestCase.java:380)
> at org.elasticsearch.test.ESIntegTestCase.setupTestCluster(ESIntegTestCase.java:2067)
> at java.lang.Thread.run(Thread.java:748)
For some reason node_s2 did not join the cluster until the test had given up on it:
1> [2018-05-25T12:24:49,797][INFO ][o.e.t.d.MockZenPing ] [node_s2] pinging using mock zen ping
1> [2018-05-25T12:25:14,794][WARN ][o.e.n.Node ] [node_s2] timed out while waiting for initial discovery state - timeout: 30s
1> [2018-05-25T12:25:14,795][INFO ][o.e.n.Node ] [node_s2] started
1> [2018-05-25T12:25:44,851][INFO ][o.s.s.s.SMTPServer ] SMTP server *:41967 stopping
1> [2018-05-25T12:25:52,165][INFO ][o.s.s.s.ServerThread ] SMTP server *:41967 stopped
1> [2018-05-25T12:25:52,171][INFO ][o.e.x.w.a.e.EmailAttachmentTests] [#testThatEmailAttachmentsAreSent]: clearing watcher state
1> [2018-05-25T12:25:52,178][INFO ][o.e.x.w.a.e.EmailAttachmentTests] waiting to stop watcher, current states [Tuple [v1=node_s2, v2=STARTED]]
1> [2018-05-25T12:25:52,186][INFO ][o.e.x.w.WatcherService ] [node_s0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
1> [2018-05-25T12:25:52,190][INFO ][o.e.x.w.WatcherService ] [node_s1] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
1> [2018-05-25T12:25:52,191][INFO ][o.e.c.s.MasterService ] [node_s1] zen-disco-node-join, reason: added {{node_s2}{FikzomkMTTueLareWsmHlQ}{AjalmjLQQReu3dFLp3FG9w}{127.0.0.1}{127.0.0.1:46204}{xpack.installed=true},}
1> [2018-05-25T12:25:52,193][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s2}{FikzomkMTTueLareWsmHlQ}{AjalmjLQQReu3dFLp3FG9w}{127.0.0.1}{127.0.0.1:46204}{xpack.installed=true},}, reason: apply cluster state (from master [master {node_s1}{T-N7bqCeQwyFLg26owJxRw}{g5GAMmHqQHSHWLyjMu_q_g}{127.0.0.1}{127.0.0.1:37292}{xpack.installed=true} committed version [8]])