Skip to content

ES|QL: Late materialization after TopN (Node level)#132757

Merged
GalLalouche merged 83 commits intoelastic:mainfrom
GalLalouche:testing_fetch_v2_passing
Oct 13, 2025
Merged

ES|QL: Late materialization after TopN (Node level)#132757
GalLalouche merged 83 commits intoelastic:mainfrom
GalLalouche:testing_fetch_v2_passing

Conversation

@GalLalouche
Copy link
Contributor

@GalLalouche GalLalouche commented Aug 12, 2025

This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase.

For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to #129454).

There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.

@GalLalouche GalLalouche requested a review from nik9000 August 19, 2025 13:24
Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with it. Let's get Alex's last few comments solved and bring this thing in for a landing. We should get some rally benchmarks out of this. It's been a lot of work. We might get this for free from the nightly, but once you are good and ready to click that merge button I think you should try and find the rally tracks that we run that'll benefit from this. So we can watch them.

Copy link
Contributor Author

@GalLalouche GalLalouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great in depth review, @alex-spies! I've addressed all things, although there is still the issue of of estimateRowSize which is waiting for @nik9000's feedback, and the questions of the dependence between the feature flags (runOnNodeReduce and reduceLateMaterialization or whatever we call it).

);
for (String q : queries) {
QueryPragmas pragmas = randomPragmas();
var pragmas = randomPragmas();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nik9000 FYI. I remember we discussed this, though I don't remember the exact solution we agreed on (if we did).


FROM dense_vector
| EVAL k = v_l2_norm(bit_vector, [1]) // workaround to enable fetching dense_vector
| EVAL k = v_l2_norm(bit_vector, [1,2]) // workaround to enable fetching dense_vector
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #136365.

@GalLalouche GalLalouche enabled auto-merge (squash) October 12, 2025 11:34
@alex-spies alex-spies self-assigned this Oct 13, 2025
@GalLalouche GalLalouche merged commit 0a7d113 into elastic:main Oct 13, 2025
34 checks passed
georgewallace pushed a commit to georgewallace/elasticsearch that referenced this pull request Oct 13, 2025
This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase.

For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to elastic#129454).

There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.
Kubik42 pushed a commit to Kubik42/elasticsearch that referenced this pull request Oct 16, 2025
This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase.

For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to elastic#129454).

There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.
GalLalouche added a commit that referenced this pull request Dec 13, 2025
…9397)

This PR fixes a slowness introduced in #132757 that was first encountered on our nightly benchmarks. After much digging, and with @nik9000's invaluable help, the culprit was found to be the usage of what is apparently a very slow functional Stream in what is apparently a very hot path.
parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this pull request Dec 17, 2025
…stic#139397)

This PR fixes a slowness introduced in elastic#132757 that was first encountered on our nightly benchmarks. After much digging, and with @nik9000's invaluable help, the culprit was found to be the usage of what is apparently a very slow functional Stream in what is apparently a very hot path.
GalLalouche added a commit that referenced this pull request Feb 23, 2026
sidosera pushed a commit to sidosera/elasticsearch that referenced this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >feature Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants