[CI] RemoteClusterClientTests testEnsureWeReconnect failing with NoSuchRemoteClusterException

[mark-vieira]

This test has failed 4 times in the past three days after passing basically 100% of the time for the past month. Look suspicious. Happening on both master and 7.x.

:server:test » org.elasticsearch.transport.RemoteClusterClientTests » testEnsureWeReconnect (1.662s)
org.elasticsearch.transport.NoSuchRemoteClusterException: no such remote cluster: [test]
java.util.concurrent.ExecutionException: org.elasticsearch.transport.NoSuchRemoteClusterException: no such remote cluster: [test]Open stacktrace
Caused by: org.elasticsearch.transport.NoSuchRemoteClusterException: no such remote cluster: [test]Open stacktrace
[2020-02-07T11:10:26,252][INFO ][o.e.t.RemoteClusterClientTests] [testEnsureWeReconnect] before test
[2020-02-07T11:10:26,808][INFO ][o.e.t.TransportService   ] [testEnsureWeReconnect] publish_address {127.0.0.1:14000}, bound_addresses {[::1]:14000}, {127.0.0.1:14000}
[2020-02-07T11:10:27,048][INFO ][o.e.t.TransportService   ] [testEnsureWeReconnect] publish_address {127.0.0.1:14001}, bound_addresses {[::1]:14001}, {127.0.0.1:14001}
[2020-02-07T11:10:27,519][INFO ][o.e.t.RemoteClusterClientTests] [testEnsureWeReconnect] after test
REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.transport.RemoteClusterClientTests.testEnsureWeReconnect" -Dtests.seed=EE4961D2D47604F4 -Dtests.security.manager=true -Dtests.locale=zh-Hans-SG -Dtests.timezone=Asia/Pontianak -Dcompiler.java=13

This same time period has also seen a rather significant jump in average test execution times so perhaps there is something going on here.

https://gradle-enterprise.elastic.co/scans/tests?failures.failureClassification=non_verification&list.offset=0&list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1581055741639&search.startTimeMin=1580450941631&search.tags=CI&search.tags=not:nested&search.tags=not:pull-request&tests.container=org.elasticsearch.transport.RemoteClusterClientTests&tests.sortField=FAILED&tests.test=testEnsureWeReconnect&tests.unstableOnly&trends.section=overview&trends.timeResolution=day&viewer.tzOffset=-480

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Network)

[albertzaharovits]

two more such failures on master
https://gradle-enterprise.elastic.co/s/gjthqdvfktvvg
https://gradle-enterprise.elastic.co/s/bp7nr62mxmski
Going to mute

[albertzaharovits]

Only muted in master d4c609b

[pugnascotia]

Failed on 7.6 too - https://gradle-enterprise.elastic.co/s/p6yxu3p22ni52