CcrRetentionLeaseIT testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused fails on CI

[tlrx]

CcrRetentionLeaseIT testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused failed yesterday on CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/2048

It does not reproduce on master with:

./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=8046052DD1D0D2FA \
  -Dtests.class=org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT \
  -Dtests.method="testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-BA \
  -Dtests.timezone=America/Campo_Grande \
  -Dcompiler.java=11 \
  -Druntime.java=8

The job execution log is
consoleText.txt

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

/cc @jasontedor

[gwbrown]

This failed again today in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+internalClusterTest/1493/console

Stack trace from this one is:

java.lang.AssertionError: 
Expected: <1551428872734L>
     but: was <1551428883142L>
	at __randomizedtesting.SeedInfo.seed([6DC9A98E216D66A5:8F62725D0881508E]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.lambda$testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused$11(CcrRetentionLeaseIT.java:712)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:868)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:842)
	at org.elasticsearch.xpack.ccr.CcrRetentionLeaseIT.testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused(CcrRetentionLeaseIT.java:693)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

Looks like this failure may have caused a bunch of errors in other tests in the same class (or maybe the other way around?), as other tests in this class fail with:

MasterNotDiscoveredException[NodeNotConnectedException[[leader0][127.0.0.1:35801] Node not connected]]; nested: NodeNotConnectedException[[leader0][127.0.0.1:35801] Node not connected];
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259)
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322)
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249)
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:549)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: NodeNotConnectedException[[leader0][127.0.0.1:35801] Node not connected]
	at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151)
	at org.elasticsearch.test.transport.StubbableConnectionManager.getConnection(StubbableConnectionManager.java:91)
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:557)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:219)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onNewClusterState(TransportMasterNodeAction.java:247)
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:308)
	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:166)
	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:117)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retry(TransportMasterNodeAction.java:243)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.access$400(TransportMasterNodeAction.java:141)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.handleException(TransportMasterNodeAction.java:229)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118)
	at org.elasticsearch.transport.TransportService$8.run(TransportService.java:960)
	... 4 more

[hub-cap]

same here as the symptoms with Node not connected and cascading failures.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+internalClusterTest/2020/console