[CI] SplitIndexIT testSplitFromOneToN failing

[luigidellaquila]

Failed only once this week.
The failure seems to be deterministic with this specific seed

Build scan:
https://gradle-enterprise.elastic.co/s/wb6ejiovyfz4g/tests/:server:internalClusterTest/org.elasticsearch.action.admin.indices.create.SplitIndexIT/testSplitFromOneToN

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitFromOneToN" -Dtests.seed=BA5CAFA3B226BD4 -Dtests.locale=en-PH -Dtests.timezone=America/Indiana/Vincennes -Druntime.java=17

Applicable branches:
8.2

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.action.admin.indices.create.SplitIndexIT&tests.test=testSplitFromOneToN

Failure excerpt:

java.lang.AssertionError: ResizeResponse failed - index creation acked but not all shards were started

  at __randomizedtesting.SeedInfo.seed([BA5CAFA3B226BD4:C1506678BE77E9E6]:0)
  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:113)
  at org.elasticsearch.action.admin.indices.create.SplitIndexIT.splitToN(SplitIndexIT.java:186)
  at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitFromOneToN(SplitIndexIT.java:87)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

I've tracked down the problem with this one, but we probably ought to transfer it to the appropriate team for a fix. The problem happens if in the test you get these variable values:

useRouting = true
useNested = true
useMixedRouting = false
useRoutingPartition = true
index.routing_partition_size = 1

In that case the shards won't be assigned because of this stack trace:

Caused by: [first_split] org.elasticsearch.action.RoutingMissingException: routing is required for [first_split]/[0]
	at org.elasticsearch.cluster.routing.IndexRouting$IdAndRoutingOnly.checkRoutingRequired(IndexRouting.java:181)
	at org.elasticsearch.cluster.routing.IndexRouting$IdAndRoutingOnly.getShard(IndexRouting.java:175)
	at org.elasticsearch.index.shard.ShardSplittingQuery$1.lambda$scorer$0(ShardSplittingQuery.java:78)
	at org.elasticsearch.index.shard.ShardSplittingQuery.findSplitDocs(ShardSplittingQuery.java:204)
	at org.elasticsearch.index.shard.ShardSplittingQuery$1.scorer(ShardSplittingQuery.java:132)
	at org.apache.lucene.index.FrozenBufferedUpdates.applyQueryDeletes(FrozenBufferedUpdates.java:399)
	at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:176)
	at org.apache.lucene.index.IndexWriter.forceApply(IndexWriter.java:5961)
	at org.apache.lucene.index.IndexWriter.tryApply(IndexWriter.java:5861)
	at org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$10(IndexWriter.java:2772)
	at org.apache.lucene.index.IndexWriter$EventQueue.processEventsInternal(IndexWriter.java:328)
	at org.apache.lucene.index.IndexWriter$EventQueue.processEvents(IndexWriter.java:317)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5704)
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3448)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3871)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3833)
	at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:204)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133)
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:462)
	... 9 more

And it looks like the reason is that we have mixed expectations when index.routing_partition_size is 1. IndexMetadata.isRoutingPartitionedIndex() returns false if that is 1, so ShardSplittingQuery thinks it is OK to pass null as the routing. But then it blows up in IndexRouting because it thinks that routing is required if the mapping requires it.
I think that the bug is that ShardSplittingQuery thinks that no routing partitioning equals no routing, and so the fix ought to go in there. But I'm not at all familiar with this code.

[masseyke]

I'm routing this to the analytics team because I think that #77731 is the commit that broke this (although I haven't checked whether the previous code had the same bug). Let me know if that's not the right team.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)