[CI] IndexShardTests.testSegmentMemoryTrackedWithRandomSearchers fails on 6.x

[dakrone]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-windows-compatibility/387/console

FAILURE 2.95s J1 | IndexShardTests.testSegmentMemoryTrackedWithRandomSearchers <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <20208L>
   >      but: was <18290L>
   > 	at __randomizedtesting.SeedInfo.seed([BAAC3BEF7A6B8157:9D2E4F2D3B8D67B0]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.index.shard.IndexShardTests.testSegmentMemoryTrackedWithRandomSearchers(IndexShardTests.java:2729)
   > 	at java.lang.Thread.run(Thread.java:748)

Which I was NOT able to reproduce this. The line from the logs however, is:

gradle :core:test -Dtests.seed=BAAC3BEF7A6B8157 -Dtests.class=org.elasticsearch.index.shard.IndexShardTests -Dtests.method="testSegmentMemoryTrackedWithRandomSearchers" -Dtests.security.manager=true -Dtests.locale=pl-PL -Dtests.timezone=Asia/Macau

[dimitris-athanasiou]

Another like this: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos/1763/consoleFull

[mayya-sharipova]

The test still fails on master multijob-darwin-compatibility.
Last failure Feb12: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/1676/consoleFull

[dakrone]

Okay, it failed from this:

  1> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
  1> 	at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_162]
  1> 	at java.util.ArrayList.remove(ArrayList.java:496) ~[?:1.8.0_162]
  1> 	at java.util.Collections$SynchronizedList.remove(Collections.java:2426) ~[?:1.8.0_162]
  1> 	at org.elasticsearch.index.shard.IndexShardTests.lambda$testSegmentMemoryTrackedWithRandomSearchers$51(IndexShardTests.java:2929) ~[test/:?]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

Because of this:

                        if (randomBoolean() && searchers.size() > 1) {
                            // Close one of the searchers at random
                            Engine.Searcher searcher = searchers.remove(0);

Since there are multiple threads if they both pull a searcher one will end up pulling where there are no searchers. I'll fix this.

[mayya-sharipova]

@dakrone thanks Lee