ES|QL: Late materialization after TopN (Node level)#132757
ES|QL: Late materialization after TopN (Node level)#132757GalLalouche merged 83 commits intoelastic:mainfrom
Conversation
…sticsearch into testing_fetch_v2_passing
…sticsearch into testing_fetch_v2_passing
…sticsearch into testing_fetch_v2_passing
nik9000
left a comment
There was a problem hiding this comment.
I'm happy with it. Let's get Alex's last few comments solved and bring this thing in for a landing. We should get some rally benchmarks out of this. It's been a lot of work. We might get this for free from the nightly, but once you are good and ready to click that merge button I think you should try and find the rally tracks that we run that'll benefit from this. So we can watch them.
...in/esql/compute/src/test/java/org/elasticsearch/compute/operator/topn/TopNOperatorTests.java
Show resolved
Hide resolved
...in/esql/compute/src/test/java/org/elasticsearch/compute/operator/topn/TopNOperatorTests.java
Show resolved
Hide resolved
...ClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlReductionLateMaterializationIT.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java
Outdated
Show resolved
Hide resolved
GalLalouche
left a comment
There was a problem hiding this comment.
Thanks for the great in depth review, @alex-spies! I've addressed all things, although there is still the issue of of estimateRowSize which is waiting for @nik9000's feedback, and the questions of the dependence between the feature flags (runOnNodeReduce and reduceLateMaterialization or whatever we call it).
...ClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlReductionLateMaterializationIT.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeSearchContext.java
Outdated
Show resolved
Hide resolved
...in/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeSearchContextByShardId.java
Outdated
Show resolved
Hide resolved
...in/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeSearchContextByShardId.java
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/DataNodeRequest.java
Outdated
Show resolved
Hide resolved
...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/LateMaterializationPlanner.java
Outdated
Show resolved
Hide resolved
...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/LateMaterializationPlanner.java
Show resolved
Hide resolved
...ck/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/DataNodeComputeHandler.java
Show resolved
Hide resolved
...in/esql/compute/src/test/java/org/elasticsearch/compute/operator/topn/TopNOperatorTests.java
Show resolved
Hide resolved
...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/LateMaterializationPlanner.java
Outdated
Show resolved
Hide resolved
| ); | ||
| for (String q : queries) { | ||
| QueryPragmas pragmas = randomPragmas(); | ||
| var pragmas = randomPragmas(); |
There was a problem hiding this comment.
@nik9000 FYI. I remember we discussed this, though I don't remember the exact solution we agreed on (if we did).
|
|
||
| FROM dense_vector | ||
| | EVAL k = v_l2_norm(bit_vector, [1]) // workaround to enable fetching dense_vector | ||
| | EVAL k = v_l2_norm(bit_vector, [1,2]) // workaround to enable fetching dense_vector |
This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase. For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to elastic#129454). There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.
This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase. For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to elastic#129454). There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.
…stic#139397) This PR fixes a slowness introduced in elastic#132757 that was first encountered on our nightly benchmarks. After much digging, and with @nik9000's invaluable help, the culprit was found to be the usage of what is apparently a very slow functional Stream in what is apparently a very hot path.
elastic#142834) Just what it says on the tin. Follow-up to elastic#141082 and elastic#132757.
This PR adds a late(r) materialization for TopN queries, such that the materialization happes in the "node_reduce" phase instead of during the "data" phase.
For example, if the limit is 20, and each data node spawns 10 workers, we would only read 20 additional columns (i.e., ones not needed for the TopN) filters, instead of 200. To support this, the reducer node maintains a global list of all shard contexts used by its individual data workers (although some of those might be closed if they are no longer needed, thanks to #129454).
There is some additional book-keeping involved, since previously, every data node held a local list of shard contexts, and used its local indices to access it. To avoid changing too much (this local-index logic is spread throughout much of the code!), a new global index is introduced, which replaces the local index after all the rows are merged together in the reduce phase's TopN.