Skip to content

Support partial filter push down #3850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
penghuo merged 11 commits into from
Jul 18, 2025
Merged

Support partial filter push down #3850

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
60cd893
Support partial filter push down
qianheng-aws Jul 4, 2025
996a82b
Add doc for PushDownAction
qianheng-aws Jul 4, 2025
b23d7ce
Fix IT
qianheng-aws Jul 4, 2025
f55ca70
Refine code to only keep non-push-down condition in the new filter
qianheng-aws Jul 7, 2025
a1159e1
Refine code
qianheng-aws Jul 9, 2025
1c7e1f5
Only show the pushed conditions in the PushDownContext
qianheng-aws Jul 14, 2025
0ee1767
Merge remote-tracking branch 'refs/remotes/origin/main' into PartialP…
qianheng-aws Jul 14, 2025
86ed34a
Ignore test when push down disabled
qianheng-aws Jul 14, 2025
ac0c48d
Fix IT after merging main
qianheng-aws Jul 15, 2025
456891b
Merge remote-tracking branch 'refs/remotes/origin/main' into PartialP…
qianheng-aws Jul 18, 2025
4649db8
Fix IT because mapping changed after merging main
qianheng-aws Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,16 @@ public void supportSearchSargPushDown_timeRange() throws IOException {
+ "| where birthdate >= '2016-12-08 00:00:00.000000000' "
+ "and birthdate < '2018-11-09 00:00:00.000000000' "));
}

// Only for Calcite
@Test
public void supportPartialPushDown() throws IOException {
// field `address` is text type without keyword subfield, so we cannot push it down.
String query =
"source=opensearch-sql_test_index_account | where age >= 1 and address = '880 Holmes Lane'"
+ " | fields age, address";
var result = explainQueryToString(query);
String expected = loadFromFile("expectedOutput/calcite/explain_partial_filter_push.json");
assertJsonEqualsIgnoreId(expected, result);
}
}
6 changes: 6 additions & 0 deletions integ-test/src/test/resources/expectedOutput/calcite/explain_partial_filter_push.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"calcite": {
"logical": "LogicalProject(age=[$8], address=[$2])\n LogicalFilter(condition=[AND(>=($8, 1), =($2, '880 Holmes Lane'))])\n CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",
"physical": "EnumerableCalc(expr#0..1=[{inputs}], expr#2=['880 Holmes Lane':VARCHAR], expr#3=[=($t0, $t2)], age=[$t1], address=[$t0], $condition=[$t3])\n CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[PROJECT->[address, age], FILTER->AND(>=($1, 1), =($0, '880 Holmes Lane'))], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"timeout\":\"1m\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"age\":{\"from\":1,\"to\":null,\"include_lower\":true,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[\"address\",\"age\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"

@LantaoJin LantaoJin Jul 11, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't be FILTER->[>=($1, 1)] instead of FILTER->AND(>=($1, 1), =($0, '880 Holmes Lane')) in PushDownContext?

@qianheng-aws qianheng-aws Jul 14, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It uses the original condition currently as its digest now. If we needs to use the pushed condition as its digest, we need to store that RexNode as well like non-pushed condition. Both is ok for functionality, the latter one should be more appropriate for explanation.

Will make that change.

}
}
7 changes: 4 additions & 3 deletions ...in/java/org/opensearch/sql/opensearch/planner/physical/OpenSearchFilterIndexScanRule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import java.util.function.Predicate;
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.plan.RelRule;
import org.apache.calcite.rel.AbstractRelNode;
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.logical.LogicalFilter;
import org.immutables.value.Value;
Expand Down Expand Up @@ -37,9 +38,9 @@ public void onMatch(RelOptRuleCall call) {
}

protected void apply(RelOptRuleCall call, Filter filter, CalciteLogicalIndexScan scan) {
CalciteLogicalIndexScan newScan = scan.pushDownFilter(filter);
if (newScan != null) {
call.transformTo(newScan);
AbstractRelNode newRel = scan.pushDownFilter(filter);
if (newRel != null) {
call.transformTo(newRel);
}
}

Expand Down
Loading
Loading