[CI] DuelScrollIT.testDuelIndexOrderQueryThenFetch

[albertzaharovits]

Build failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/1225/console

Does not reproduce:

./gradlew :server:integTest \
  -Dtests.seed=74DD3997657C556E \
  -Dtests.class=org.elasticsearch.search.scroll.DuelScrollIT \
  -Dtests.method="testDuelIndexOrderQueryThenFetch" \
  -Dtests.security.manager=true \
  -Dtests.locale=nn-NO \
  -Dtests.timezone=Africa/Ceuta

Relevant log excerpt:

06:35:47   1> {"took":24,"timed_out":false,"num_reduce_phases":4,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"_clusters":{"total":0,"successful":0,"skipped":0},"hits":{"total":19,"max_score":0.61310446,"hits":[{"_index":"test","_type":"type","_id":"26","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"33","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"4","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"5","_score":0.61310446,"_source":{"foo":true},"sort":[1]},{"_index":"test","_type":"type","_id":"29","_score":0.61310446,"_source":{"foo":true},"sort":[2]},{"_index":"test","_type":"type","_id":"0","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"15","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"1","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"22","_score":0.61310446,"_source":{"foo":true},"sort":[4]},{"_index":"test","_type":"type","_id":"27","_score":0.61310446,"_source":{"foo":true},"sort":[4]},{"_index":"test","_type":"type","_id":"24","_score":0.61310446,"_source":{"foo":true},"sort":[5]},{"_index":"test","_type":"type","_id":"3","_score":0.61310446,"_source":{"foo":true},"sort":[5]},{"_index":"test","_type":"type","_id":"25","_score":0.61310446,"_source":{"foo":true},"sort":[6]},{"_index":"test","_type":"type","_id":"10","_score":0.61310446,"_source":{"foo":true},"sort":[6]},{"_index":"test","_type":"type","_id":"16","_score":0.61310446,"_source":{"foo":true},"sort":[6]},{"_index":"test","_type":"type","_id":"23","_score":0.61310446,"_source":{"foo":true},"sort":[6]},{"_index":"test","_type":"type","_id":"34","_score":0.61310446,"_source":{"foo":true},"sort":[7]},{"_index":"test","_type":"type","_id":"20","_score":0.61310446,"_source":{"foo":true},"sort":[7]},{"_index":"test","_type":"type","_id":"8","_score":0.61310446,"_source":{"foo":true},"sort":[8]}]}}
06:35:47   1> [2018-08-07T08:35:46,687][INFO ][o.e.s.s.DuelScrollIT     ] [testDuelIndexOrderQueryThenFetch] Scroll size=10, from=0:
06:35:47   1> {"_scroll_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAAlFjFnUU1NWURpVERlV1d2MlhmeExUWFEAAAAAAAAAJhYxZ1FNTVlEaVREZVdXdjJYZnhMVFhRAAAAAAAAACMWMWdRTU1ZRGlURGVXV3YyWGZ4TFRYUQAAAAAAAAAkFjFnUU1NWURpVERlV1d2MlhmeExUWFEAAAAAAAAAIhYxZ1FNTVlEaVREZVdXdjJYZnhMVFhR","took":35,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"_clusters":{"total":0,"successful":0,"skipped":0},"hits":{"total":19,"max_score":0.61310446,"hits":[{"_index":"test","_type":"type","_id":"26","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"33","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"4","_score":0.61310446,"_source":{"foo":true},"sort":[0]},{"_index":"test","_type":"type","_id":"5","_score":0.61310446,"_source":{"foo":true},"sort":[1]},{"_index":"test","_type":"type","_id":"29","_score":0.61310446,"_source":{"foo":true},"sort":[2]},{"_index":"test","_type":"type","_id":"0","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"15","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"1","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"3","_score":0.61310446,"_source":{"foo":true},"sort":[3]},{"_index":"test","_type":"type","_id":"22","_score":0.61310446,"_source":{"foo":true},"sort":[4]}]}}
06:35:47   1> [2018-08-07T08:35:46,689][INFO ][o.e.s.s.DuelScrollIT     ] [testDuelIndexOrderQueryThenFetch] [DuelScrollIT#testDuelIndexOrderQueryThenFetch]: cleaning up after test
06:35:47   1> [2018-08-07T08:35:46,718][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/BPc3XyYuQJOAwDEoshowKw] deleting index
06:35:47   1> [2018-08-07T08:35:46,907][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template]
06:35:47   1> [2018-08-07T08:35:46,919][INFO ][o.e.s.s.DuelScrollIT     ] [testDuelIndexOrderQueryThenFetch] [DuelScrollIT#testDuelIndexOrderQueryThenFetch]: cleaned up after test
06:35:47   1> [2018-08-07T08:35:46,920][INFO ][o.e.s.s.DuelScrollIT     ] [testDuelIndexOrderQueryThenFetch] after test
06:35:47 FAILURE 1.45s J2 | DuelScrollIT.testDuelIndexOrderQueryThenFetch <<< FAILURES!
06:35:47   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+multijob-unix-compatibility/os/oraclelinux/server/build/testrun/integTest/J2/temp/org.elasticsearch.search.scroll.DuelScrollIT_74DD3997657C556E-001
06:35:47   2> NOTE: test params are: codec=Asserting(Lucene70): {_routing=PostingsFormat(name=Asserting), nested.field4=PostingsFormat(name=Asserting), _field_names=PostingsFormat(name=Asserting), foo=PostingsFormat(name=Asserting), _type=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), field2=PostingsFormat(name=Asserting)}, docValues:{nested.field3=DocValuesFormat(name=Lucene70), _routing=DocValuesFormat(name=Lucene70), _seq_no=DocValuesFormat(name=Asserting), nested.field4=DocValuesFormat(name=Lucene70), field1=DocValuesFormat(name=Asserting), _field_names=DocValuesFormat(name=Asserting), foo=DocValuesFormat(name=Lucene70), _source=DocValuesFormat(name=Asserting), _id=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Lucene70), field2=DocValuesFormat(name=Lucene70), _version=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=979, maxMBSortInHeap=7.993940879650824, sim=RandomSimilarity(queryNorm=true): {}, locale=nn-NO, timezone=Africa/Ceuta
06:35:47   2> NOTE: Linux 3.10.0-693.el7.x86_64 amd64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=16,threads=1,free=378472808,total=514850816
06:35:47   2> NOTE: All tests run in this JVM: [DuelScrollIT]
06:35:47    > Throwable #1: org.junit.ComparisonFailure: expected:<[22]> but was:<[3]>
06:35:47    > 	at __randomizedtesting.SeedInfo.seed([74DD3997657C556E:7C984237C6A4F460]:0)
06:35:47    > 	at org.elasticsearch.search.scroll.DuelScrollIT.testDuelIndexOrder(DuelScrollIT.java:244)
06:35:47    > 	at org.elasticsearch.search.scroll.DuelScrollIT.testDuelIndexOrderQueryThenFetch(DuelScrollIT.java:264)
06:35:47    > 	at java.lang.Thread.run(Thread.java:748)
06:35:47   1> [2018-08-07T08:35:46,941][INFO ][o.e.n.Node               ] [suite] stopping ...
06:35:47   1> [2018-08-07T08:35:46,954][INFO ][o.e.n.Node               ] [suite] stopped

[elasticmachine]

Pinging @elastic/es-search-aggs

[alpar-t]

Failed on master too: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/226/

16:12:07 FAILURE 1.96s J3 | DuelScrollIT.testDuelIndexOrderQueryThenFetch <<< FAILURES!
16:12:07    > Throwable #1: org.junit.ComparisonFailure: expected:<[51]> but was:<[62]>
16:12:07    > 	at __randomizedtesting.SeedInfo.seed([71E44200CDA65CF2:79A139A06E7EFDFC]:0)
16:12:07    > 	at org.elasticsearch.search.scroll.DuelScrollIT.testDuelIndexOrder(DuelScrollIT.java:244)
16:12:07    > 	at org.elasticsearch.search.scroll.DuelScrollIT.testDuelIndexOrderQueryThenFetch(DuelScrollIT.java:263)
16:12:07    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:12:07    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:12:07    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:12:07    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
16:12:07    > 	at java.base/java.lang.Thread.run(Thread.java:844)

Tried and failed to reproduce with:

[:~/work/elastic/elasticsearch] 781e6ad5517+ ± ./gradlew :server:integTest -Dtests.seed=71E44200CDA65CF2 -Dtests.class=org.elasticsearch.search.scroll.DuelScrollIT -Dtests.method="testDuelIndexOrderQueryThenFetch" -Dtests.security.manager=true -Dtests.locale=sr-BA -Dtests.timezone=Mexico/General

[colings86]

Failed again on 6.x here: http://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/2266

I will mute on master and 6.x

[colings86]

Muted on master in d80457e and 6.x in 08936d4

[colings86]

@jimczi would you be able to look into this one?

[jimczi]

I pushed a fix, this was due to the recent Lucene 7.5 upgrade which explains why the test does not fail on es 6.4.