GatewayIndexStateIT.testIndexDeletionWhenNodeRejoins failure

[jpountz]

This failure does not reproduce at all for me, and might be specific to Windows. I have not disabled the test as this looks like a rare failure.

http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/5235/console

REPRODUCE WITH: gradle :core:integTest -Dtests.seed=B50FA50310ECA5F2 -Dtests.class=org.elasticsearch.gateway.GatewayIndexStateIT -Dtests.nightly=false -Dtests.method="testIndexDeletionWhenNodeRejoins" -Dtests.security.manager=true -Dtests.nightly=false -Dtests.heap.size=1024m -Dtests.jvm.argline="-server -XX:+UseConcMarkSweepGC -XX:+UseCompressedOops -XX:+AggressiveOpts" -Dtests.locale=es-PA -Dtests.timezone=Africa/Tunis

FAILURE 11.1s J0 | GatewayIndexStateIT.testIndexDeletionWhenNodeRejoins <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 	at __randomizedtesting.SeedInfo.seed([B50FA50310ECA5F2:99E39DB472A57CD6]:0)
   > 	at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:715)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:689)
   > 	at org.elasticsearch.gateway.GatewayIndexStateIT.testIndexDeletionWhenNodeRejoins(GatewayIndexStateIT.java:383)
   > 	at java.lang.Thread.run(Thread.java:745)
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: index folder sWraMKhBTdmYMIIOp39SIw should be deleted
   > 		at org.elasticsearch.gateway.GatewayIndexStateIT.lambda$testIndexDeletionWhenNodeRejoins$1128(GatewayIndexStateIT.java:386)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:703)
   > 		... 39 more

[jpountz]

@s1monw I assigned it to you because I thought you would know who would be the right person to look at it.

[bleskes]

@abeyad can you maybe look at this?

[abeyad]

This test uses a randomBoolean() to either run the test with a regular index or with a shadow replica index. This test run failed on a shadow replica index. I'm thinking Windows is holding onto a file lock for the shadow replica and hence the index directory can not be deleted. Regardless, since we are removing support for shadow replicas in #23906 and that PR already removes the shadow replica line of execution in https://github.com/elastic/elasticsearch/pull/23906/files#diff-8bda2c6e1d5d51d8e550a445373f93e5L333, I think we can just close this for the time being?

[abeyad]

I do think we should remove the shadow replica option for that test in the 5.x branch as well, since shadow replicas are no longer supported.

[bleskes]

+1

…

On 10 Apr 2017, at 15:23, Ali Beyad ***@***.***> wrote: I do think we should remove the shadow replica option for that test in the 5.x branch as well. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[abeyad]

fixed by 94aa8f4

[martijnvg]

The test failed again, but this time differently:

Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=13159, name=elasticsearch[node_t1][generic][T#1], state=RUNNABLE, group=TGRP-GatewayIndexStateIT]
   > 	at __randomizedtesting.SeedInfo.seed([D87704880C55588B:F49B3C3F6E1C81AF]:0)
   > Caused by: org.apache.lucene.store.AlreadyClosedException: engine is closed
   > 	at __randomizedtesting.SeedInfo.seed([D87704880C55588B]:0)
   > 	at org.elasticsearch.index.shard.IndexShard.getEngine(IndexShard.java:1650)
   > 	at org.elasticsearch.index.shard.IndexShard.failShard(IndexShard.java:951)
   > 	at org.elasticsearch.index.shard.IndexShard.lambda$updatePrimaryTerm$1(IndexShard.java:376)
   > 	at org.elasticsearch.index.shard.IndexShardOperationPermits$1.onFailure(IndexShardOperationPermits.java:124)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623)
   > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   > 	at java.lang.Thread.run(Thread.java:748)

Link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/3020/console