FullClusterRestartIT#testRecovery() fails in CI

[danielmitterdorfer]

build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.1+multijob-unix-compatibility/os=centos/66/console

reproduction line:

gradle :qa:full-cluster-restart:v6.0.2-SNAPSHOT#upgradedClusterTestRunner \
  -Dtests.seed=FEEAF874546C710B \
  -Dtests.class=org.elasticsearch.upgrades.FullClusterRestartIT \
  -Dtests.method="testRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=ro \
  -Dtests.timezone=America/Argentina/Jujuy

failure message:

java.lang.AssertionError: mismatch while checking for translog recovery
testrecovery 0 existing_store done 0
testrecovery 0 peer           done 0
testrecovery 1 existing_store done 0
testrecovery 1 peer           done 0
testrecovery 2 existing_store done 0
testrecovery 2 peer           done 0
testrecovery 3 existing_store done 0
testrecovery 3 peer           done 0
testrecovery 4 existing_store done 0
testrecovery 4 peer           done 52
 expected:<false> but was:<true>
	at __randomizedtesting.SeedInfo.seed([FEEAF874546C710B:3F1A81D8793CBBAC]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.elasticsearch.upgrades.FullClusterRestartIT.testRecovery(FullClusterRestartIT.java:669)
[...]

[nik9000]

Another:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/1718/consoleText

FAILURE 0.11s | FullClusterRestartIT.testRecovery <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: mismatch while checking for translog recovery
   > testrecovery 0 existing_store done 0
   > testrecovery 0 peer           done 0
   > testrecovery 1 existing_store done 0
   > testrecovery 1 peer           done 0
   > testrecovery 2 existing_store done 0
   > testrecovery 2 peer           done 0
   > testrecovery 3 existing_store done 0
   > testrecovery 3 peer           done 0
   > testrecovery 4 existing_store done 54
   > testrecovery 4 peer           done 0
   >  expected:<false> but was:<true>
   > 	at __randomizedtesting.SeedInfo.seed([45B4F88AE72AABD3:84448126CA7A6174]:0)
   > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testRecovery(FullClusterRestartIT.java:742)
   > 	at java.lang.Thread.run(Thread.java:748)

[javanna]

This looks like another occurrence of this failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles/2289

11:32:38 FAILURE 0.07s | FullClusterRestartIT.testRecovery <<< FAILURES!
11:32:38    > Throwable #1: java.lang.AssertionError: mismatch while checking for translog recovery
11:32:38    > testrecovery 0 existing_store done 0
11:32:38    > testrecovery 0 peer           done 0
11:32:38    > testrecovery 1 existing_store done 0
11:32:38    > testrecovery 1 peer           done 0
11:32:38    > testrecovery 2 existing_store done 0
11:32:38    > testrecovery 2 peer           done 0
11:32:38    > testrecovery 3 existing_store done 0
11:32:38    > testrecovery 3 peer           done 0
11:32:38    > testrecovery 4 existing_store done 0
11:32:38    > testrecovery 4 peer           done 21
11:32:38    >  expected:<false> but was:<true>
11:32:38    > 	at __randomizedtesting.SeedInfo.seed([DC990C8FC676707F:1D697523EB26BAD8]:0)
11:32:38    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testRecovery(FullClusterRestartIT.java:743)

[javanna]

@bleskes this issue has been opened for a while, I see you are assigned to it, is there any progress or should it be reassigned?