[CI] CCSDuelIT testFieldCollapsingOneClusterHasNoResults failing

[dimitris-athanasiou]

Build scan:
https://gradle-enterprise.elastic.co/s/me2w54x4rh5tw/tests/:qa:multi-cluster-search:mixedClusterTest/org.elasticsearch.search.CCSDuelIT/testFieldCollapsingOneClusterHasNoResults

Reproduction line:
./gradlew ':qa:multi-cluster-search:mixedClusterTest' --tests "org.elasticsearch.search.CCSDuelIT.testFieldCollapsingOneClusterHasNoResults" -Dtests.seed=590265B01FD5045C -Dtests.locale=hu -Dtests.timezone=Brazil/Acre -Druntime.java=11

Applicable branches:
master

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.search.CCSDuelIT&tests.test=testFieldCollapsingOneClusterHasNoResults

Failure excerpt:

java.lang.AssertionError: Didn't match expected value:
                     _clusters: 
                         skipped: same [0]
                      successful: same [2]
                           total: same [2]
                       _shards: 
                          failed: same [0]
                         skipped: expected Integer [5] but was Integer [0]
                      successful: same [8]
                           total: same [8]
                          hits: 
                            hits: 
                                 0: 
                                 _id: same [multi_cluster-1]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/30]
                                    id: same [multi_cluster-1]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [python]
                                         1: same [java]
                                         2: same [php]
                                  tags: 
                                       0: same [python]
                                       1: same [java]
                                       2: same [php]
                                  type: same [question]
                                  user: same [user3]
                                 votes: same [6]
                              fields: 
                          user.keyword: 
                                       0: same [user3]
                                 1: 
                                 _id: same [multi_cluster-6]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/21]
                                    id: same [multi_cluster-6]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [html]
                                  tags: 
                                       0: same [html]
                                  type: same [question]
                                  user: same [user8]
                                 votes: same [8]
                              fields: 
                          user.keyword: 
                                       0: same [user8]
                                 2: 
                                 _id: same [multi_cluster-8]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/02]
                                    id: same [multi_cluster-8]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [perl]
                                         1: same [ruby]
                                  tags: 
                                       0: same [perl]
                                       1: same [ruby]
                                  type: same [question]
                                  user: same [user4]
                                 votes: same [14]
                              fields: 
                          user.keyword: 
                                       0: same [user4]
                                 3: 
                                 _id: same [multi_cluster-13]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/30]
                                    id: same [multi_cluster-13]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [python]
                                         1: same [php]
                                  tags: 
                                       0: same [python]
                                       1: same [php]
                                  type: same [question]
                                  user: same [user7]
                                 votes: same [14]
                              fields: 
                          user.keyword: 
                                       0: same [user7]
                                 4: 
                                 _id: same [multi_cluster-23]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/26]
                                    id: same [multi_cluster-23]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [java]
                                  tags: 
                                       0: same [java]
                                  type: same [question]
                                  user: same [user5]
                                 votes: same [9]
                              fields: 
                          user.keyword: 
                                       0: same [user5]
                                 5: 
                                 _id: same [multi_cluster-28]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/15]
                                    id: same [multi_cluster-28]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [ruby]
                                  tags: 
                                       0: same [ruby]
                                  type: same [question]
                                  user: same [user1]
                                 votes: same [16]
                              fields: 
                          user.keyword: 
                                       0: same [user1]
                                 6: 
                                 _id: same [multi_cluster-31]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/19]
                                    id: same [multi_cluster-31]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [python]
                                         1: same [sql]
                                         2: same [ruby]
                                  tags: 
                                       0: same [python]
                                       1: same [sql]
                                       2: same [ruby]
                                  type: same [question]
                                  user: same [user9]
                                 votes: same [12]
                              fields: 
                          user.keyword: 
                                       0: same [user9]
                                 7: 
                                 _id: same [multi_cluster-37]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/28]
                                    id: same [multi_cluster-37]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [python]
                                         1: same [perl]
                                  tags: 
                                       0: same [python]
                                       1: same [perl]
                                  type: same [question]
                                  user: same [user6]
                                 votes: same [23]
                              fields: 
                          user.keyword: 
                                       0: same [user6]
                                 8: 
                                 _id: same [multi_cluster-55]
                              _index: same [ccs_duel_index]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/04]
                                    id: same [multi_cluster-55]
                                  join: 
                                    name: same [question]
                            questionId: same [null]
                               suggest: 
                                   input: 
                                         0: same [perl]
                                         1: same [ruby]
                                  tags: 
                                       0: same [perl]
                                       1: same [ruby]
                                  type: same [question]
                                  user: same [user2]
                                 votes: same [5]
                              fields: 
                          user.keyword: 
                                       0: same [user2]
                                 9: 
                                 _id: same [multi_cluster-1005]
                              _index: same [ccs_duel_index]
                            _routing: same [multi_cluster-30]
                              _score: same [1.0]
                             _source: 
                          creationDate: same [2019/01/20]
                                    id: same [multi_cluster-1005]
                                  join: 
                                    name: same [answer]
                                  parent: same [multi_cluster-30]
                            questionId: same [multi_cluster-30]
                               suggest: 
                                   input: same [empty list]
                                  tags: same [empty list]
                                  type: same [answer]
                                  user: same [user10]
                                 votes: same [7]
                              fields: 
                          user.keyword: 
                                       0: same [user10]
                       max_score: same [null]
                           total: 
                          relation: same [eq]
                             value: same [204]
                     timed_out: same [false]
                          took: same [-1]

  at __randomizedtesting.SeedInfo.seed([590265B01FD5045C:92C56F0B55D4C02]:0)
  at org.elasticsearch.search.CCSDuelIT.duelSearch(CCSDuelIT.java:771)
  at org.elasticsearch.search.CCSDuelIT.testFieldCollapsingOneClusterHasNoResults(CCSDuelIT.java:472)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

As far as I can see #78765 added randomization of preFilterShardSize to this test that seems to trigger this. Not sure atm of the test needs to be adapted or this shows some error elsewhere.

[cbuescher]

Muted on master and 7.x for now

[ywelsch]

The randomization of preFilterShardSize (added in #78765) is breaking this test. With the given seed, it creates the following situation:

• CCS setup
• preFilterShardSize is set to 7
• index ccs_duel_index has 5 shards in cluster 1 and 5 shards in cluster 2
• The query ranges over both indices
• when minimizeRoundtrips is set to true, each coordinator operates on 5 shards, i.e. below the preFilterShardSize of 7. No shards are skipped as the can-match phase isn't run.
• when minimizeRoundtrips is set to false, the coordinator operates on 5+5 = 10 shards, i.e. above the preFilterShardSize of 7. The can-match is run, allowing to skip 5 shards.

I'm not sure yet how I would like to fix the test, just putting the analysis of the situation here.
cc: @dnhatn