SnapshotLifecycleRestIT#testPolicyManualExecution failed

[DaveCTurner]

This build failed: https://gradle-enterprise.elastic.co/s/gxyu6m3t7am7y

org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT > testPolicyManualExecution FAILED
    java.lang.AssertionError: 
    Expected: <1>
         but: was <12>
        at __randomizedtesting.SeedInfo.seed([2DD52D02B7D3C545:96E4C1D59BA0B901]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.lambda$testPolicyManualExecution$4(SnapshotLifecycleRestIT.java:240)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:893)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:866)
        at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution(SnapshotLifecycleRestIT.java:219)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution" -Dtests.seed=2DD52D02B7D3C545 -Dtests.security.manager=true -Dtests.locale=en-IN -Dtests.timezone=Asia/Ulan_Bator -Dcompiler.java=12 -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[gwbrown]

I've actually seen this failure before, I believe it's the same root cause as #48017.

[gwbrown]

I've added some extra logging in #48612 that will hopefully help catch this failure. I'll try to keep an eye out for failures but if you find a failure for this test please ping me.

[polyfractal]

@gwbrown Not sure if this is related, but some of these SLM tests just failed again (SnapshotLifecycleRestIT.testPolicyManualExecution and SnapshotLifecycleRestIT.testBasicTimeBasedRetenion):

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/2563/

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution" \
  -Dtests.seed=BE08EA40E0D6922C \
  -Dtests.security.manager=true \
  -Dtests.locale=ar \
  -Dtests.timezone=America/Manaus \
  -Dcompiler.java=12

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testBasicTimeBasedRetenion" \
  -Dtests.seed=BE08EA40E0D6922C \
  -Dtests.security.manager=true \
  -Dtests.locale=ar \
  -Dtests.timezone=America/Manaus \
  -Dcompiler.java=12

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution" \
  -Dtests.seed=BE08EA40E0D6922C \
  -Dtests.security.manager=true \
  -Dtests.locale=ar \
  -Dtests.timezone=America/Manaus \
  -Dcompiler.java=12

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testBasicTimeBasedRetenion" \
  -Dtests.seed=BE08EA40E0D6922C \
  -Dtests.security.manager=true \
  -Dtests.locale=ar \
  -Dtests.timezone=America/Manaus \
  -Dcompiler.java=12

java.lang.AssertionError: 
Expected: <1>
     but: was <12>
	at __randomizedtesting.SeedInfo.seed([BE08EA40E0D6922C:5390697CCA5EE68]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.lambda$testPolicyManualExecution$6(SnapshotLifecycleRestIT.java:250)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:866)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:839)
	at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution(SnapshotLifecycleRestIT.java:229)

java.lang.AssertionError: 
Expected: <1>
     but: was <65>
	at __randomizedtesting.SeedInfo.seed([BE08EA40E0D6922C:2B0F8260C2409964]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.lambda$testBasicTimeBasedRetenion$13(SnapshotLifecycleRestIT.java:423)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:866)
	at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testBasicTimeBasedRetenion(SnapshotLifecycleRestIT.java:404)

Expected: a string containing "snapshot_missing_exception"
     but: was "{"responses":[{"repository":"my-repo","snapshots":[{"snapshot":"snap-nqglh02vs4acgf0blqyd7w","uuid":"Q9ySxpEXQBGe_HksIVsYyg","version_id":8000099,"version":"8.0.0","indices":["test"],"include_global_state":true,"metadata":{"policy":"test-policy"},"state":"SUCCESS","start_time":"2019-12-04T15:13:16.343Z","start_time_in_millis":1575472396343,"end_time":"2019-12-04T15:13:16.552Z","end_time_in_millis":1575472396552,"duration_in_millis":209,"failures":[],"shards":{"total":1,"failed":0,"successful":1}}]}]}"

Edit: Doesn't reproduce for me :(

[dimitris-athanasiou]

And another failure of this in https://gradle-enterprise.elastic.co/s/s6rpuelnp5lda

:x-pack:plugin:ilm:qa:multi-node:integTestRunner » org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT » testPolicyManualExecution (14.333s)

Expected: <1>
     but: was <11>
java.lang.AssertionError: 

Expected: <1>
     but: was <11>
Close stacktrace
at __randomizedtesting.SeedInfo.seed([1482194B8AF10557:AFB3F59CA6827913]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.lambda$testPolicyManualExecution$4(SnapshotLifecycleRestIT.java:241)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:900)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:873)
at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testPolicyManualExecution(SnapshotLifecycleRestIT.java:220)