Projection pushdown below Exchange + QTF adaptation - #22082
Conversation
PR Reviewer Guide 🔍(Review updated until commit 0bff4dc)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 0bff4dc Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 9438fa9
Suggestions up to commit 540429c
Suggestions up to commit c573cb2
Suggestions up to commit 5ee8f7f
Suggestions up to commit a024e2e
|
…rite accordingly Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 5eda2f3 |
|
❌ Gradle check result for 5eda2f3: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 012994b |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #22082 +/- ##
============================================
- Coverage 73.34% 73.30% -0.05%
- Complexity 75800 75806 +6
============================================
Files 6064 6064
Lines 344501 344498 -3
Branches 49575 49575
============================================
- Hits 252670 252522 -148
- Misses 71649 71859 +210
+ Partials 20182 20117 -65 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit b04ee2d |
|
❌ Gradle check result for b04ee2d: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 5acbbbd |
|
Persistent review updated to latest commit 0101411 |
|
❌ Gradle check result for 0101411: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
@expani I think the sandbox errors are related to this change: https://github.com/opensearch-project/OpenSearch/actions/runs/27251790133/job/80483842313?pr=22082 Can you please check once if any tests need modifications. |
|
Persistent review updated to latest commit 188161a |
|
Persistent review updated to latest commit a024e2e |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 5ee8f7f |
|
Persistent review updated to latest commit c573cb2 |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 540429c |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 9438fa9 |
|
❌ Gradle check result for 9438fa9: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aniketh Jain <anijainc@amazon.com>
|
Persistent review updated to latest commit 0bff4dc |
…ct#22082) * Removed Sort Project Transpose rule and handled QTF detection and rewrite accordingly Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Fixed assertion caused by removal of Sort Project Transpose Signed-off-by: Aniketh Jain <anijainc@amazon.com> * More changes to shape test because of Sort Project Transpose Removal Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Accounting for RowId and UGSI in FieldStorageInfo Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Added rule to duplicate 2 projects for percentile aggs to work Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Fixed the assertions for the new Rule Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Not dropping pure Project columns to ensure order is maintained Signed-off-by: Aniketh Jain <anijainc@amazon.com> * Added plan shape tests for the new rule and fixed issue with TAKE Signed-off-by: Aniketh Jain <anijainc@amazon.com> --------- Signed-off-by: Aniketh Jain <anijainc@amazon.com> Co-authored-by: Sandesh Kumar <sandeshkr419@gmail.com>
What
Push
Projectbelow the Exchange so DataFusion reads only referenced columns from parquet on each shard, instead of scanning all ~105 columns and projecting at the coordinator.Why
PPL queries like
source=idx | where x | fields bodywere reading every column off parquet and shipping them across the gather. Wasteful I/O and network.Changes
PlannerImpl.pushdownRules— removeSORT_PROJECT_TRANSPOSE. Keeps the SELECTProjectbelow the Sort/Exchange so DataFusion'sOptimizeProjectionsprunes the scan.OpenSearchExchangeReducer.computeSelfCost— scale cost by output column count (× getRowType().getFieldCount()). Volcano now prefers placing the Exchange where fewer columns cross it.OpenSearchLateMaterializationRewriter— adapt QTF, which previously assumedSORT_PROJECT_TRANSPOSElifted the Project above the Sort:Verified
Notes for reviewers
The QTF change is a consequence of removing
SORT_PROJECT_TRANSPOSE, not a feature — it preserves QTF behavior (same fire set, same scan narrowing, same deferred-fetch set), not improves it.