[CI] SimpleSearchIT testSimpleIndexSortEarlyTerminate failing

[romseygeek]

CheckIndex is running against an index with an index sort defined on an int field and is running into a int/long mismatch

Build scan:
https://gradle-enterprise.elastic.co/s/r7x25vzhq3ujw/tests/:server:internalClusterTest/org.elasticsearch.search.simple.SimpleSearchIT/testSimpleIndexSortEarlyTerminate

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.simple.SimpleSearchIT.testSimpleIndexSortEarlyTerminate" -Dtests.seed=CFCF164FEA77FB74 -Dtests.locale=en-IN -Dtests.timezone=Asia/Atyrau -Druntime.java=11

Applicable branches:
master

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.search.simple.SimpleSearchIT&tests.test=testSimpleIndexSortEarlyTerminate

Failure excerpt:

java.lang.AssertionError: at least one shard failed CheckIndex

  at __randomizedtesting.SeedInfo.seed([CFCF164FEA77FB74:761D17D1362954F]:0)
  at org.elasticsearch.test.ESTestCase.ensureCheckIndexPassed(ESTestCase.java:555)
  at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:390)
  at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(null:-1)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[mark-vieira]

I've muted this for now since it's still failing with some regularity.

[ywelsch]

Lucene 9 requires that the same data/type is stored in points and doc values as numeric sort uses indexed points to skip over non-competitive documents (see also https://issues.apache.org/jira/browse/LUCENE-9334). We break this assumption in ES by using the wider numeric sort type for every field (e.g. shorts use longs and floats use doubles).
#78103 works around this by forbidding the usage of points in numeric sort on field types that use a different sort type.

That only solves the issue at query time, but still looks to leave us in a place where the index is considered inconsistent from Lucene's perspective, with CheckIndex failing.

This is horrible, as it means that users running consistency checks on their index will fail, with the index being considered as corrupt.

[ywelsch]

cc: @mayya-sharipova (LUCENE-10089)

[romseygeek]

This will be fixed by LUCENE-10377, which is merged to the lucene 9.1 branch

[ywelsch]

Just a thought I had on waiting for Lucene fixes to make it into future ES versions.

What if we removed the AwaitsFix on the test and instead added

assumeTrue("bug fixed in Lucene 9.1.0, see LUCENE-10377", Version.CURRENT.luceneVersion.onOrAfter(org.apache.lucene.util.Version.fromBits(9, 1, 0)));

to the beginning of test. Then we could close this issue and forget all about it.

It adds a bit more technical debt to the testing code base (but could eventually be cleaned up), but would allow us to forget about this open issue here.

A variant of this proposal would be to add

assert Version.CURRENT.luceneVersion.onOrAfter(org.apache.lucene.util.Version.fromBits(9, 1, 0)) == false;

after the assume statement, which would make the test fail once Lucene is being upgraded, and force the PR that upgrades Lucene to clean up the technical debt by removing both statements.

WDYT?