ES|QL - LuceneTopNSourceOperator data partition strategy uses ContextIndexSearcher slicing#143133
Merged
carlosdelest merged 38 commits intoelastic:mainfrom Mar 1, 2026
Conversation
…ne-top-n-data-partition-strategy
…p-n-data-partition-strategy' into enhancement/esql-lucene-top-n-data-partition-strategy
…into enhancement/esql-topn-data-partition-strategy # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/aggregate/SumOverTimeTests.java
…-data-partition-strategy # Conflicts: # x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/query/LuceneSliceQueue.java
carlosdelest
added
>enhancement
Team:Analytics
Collaborator
|
Hi @carlosdelest, I've created a changelog YAML for you. |
carlosdelest
added
auto-backport
…-partition-strategy' into enhancement/esql-topn-data-partition-strategy
…-data-partition-strategy
Collaborator
|
Hi @carlosdelest, I've updated the changelog YAML for you. |
…-data-partition-strategy # Conflicts: # x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/query/LuceneSourceOperator.java
…-partition-strategy' into enhancement/esql-topn-data-partition-strategy
…-partition-strategy' into enhancement/esql-topn-data-partition-strategy
ioanatia
approved these changes
Feb 27, 2026
Contributor
ioanatia
left a comment
ioanatia
left a comment
There was a problem hiding this comment.
looks good, I was wondering if we can further simplify some checks now that the node level materialization is enabled on release builds
|
|
||
| private static AutoStrategy topNAutoStrategy() { | ||
| return unusedLimit -> { | ||
| if (EsqlCapabilities.Cap.ENABLE_REDUCE_NODE_LATE_MATERIALIZATION.isEnabled()) { |
Contributor
There was a problem hiding this comment.
looks like this capability is always enabled - see #142834
do we still need this check - or could we always use LuceneSliceQueue.PartitioningStrategy.SEGMENT
Member
Author
There was a problem hiding this comment.
Ooooh, that happened!
Thanks for checking, I removed the cap check in 15cd5aa
| private static Matcher<String> expectedAutoPartition(String index, String idxPartition) { | ||
| private static Matcher<String> expectedAutoPartition(String index, String idxPartition, boolean score) { | ||
| boolean lateMaterializationEnabled = EsqlCapabilities.Cap.ENABLE_REDUCE_NODE_LATE_MATERIALIZATION.isEnabled(); | ||
| if (score && lateMaterializationEnabled == false) { |
Contributor
There was a problem hiding this comment.
same here - the cap is always enabled - do we still need this check since this is a test only for single node? (so it's not used for bwc tests).
| * </ul> | ||
| */ | ||
| private static PartitioningStrategy highSpeedAutoStrategy(Query query) { | ||
| public static PartitioningStrategy highSpeedAutoStrategy(Query query) { |
Contributor
There was a problem hiding this comment.
nitpick - but do we need this to be public 🙈
carlosdelest
changed the title
Member
Author
|
@elasticsearchmachine run elasticsearch-ci/part-3 |
Member
Author
|
@elasticsearchmachine run elasticsearch-ci/part-3 |
tballison
pushed a commit
to tballison/elasticsearch
that referenced
this pull request
Mar 3, 2026
…IndexSearcher slicing (elastic#143133)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related #141770
#142128 used high speed partitioning for LuceneTopNSourceOperator. That improved p90 timing but worsened p50 timing, and was reverted in #142453.
Analyzing the Lucene query execution differences, it was clear that ESQL used more slices than the DSL. An excessive number of slices was harming the average use case.
The
SEGMENTdata partitioning strategy used the LuceneIndexSearcher.slices()static method for calculating the slices. This is not what the DSL does, as it uses the ContextIndexSearcher that in turn overrides some of the default Lucene behavior.This PR modifies the
SEGMENTstrategy by using theIndexSearcherthat it receives as an instance to calculate the slices via thegetSlices()method, instead of using the static IndexSearcher method from Lucene.This is consistent with the DSL and provides an upper limit on the number of slices created based on the available search threads available.