Skip to content

Support full expression in WHERE clauses #3849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
penghuo merged 8 commits into from
Jul 21, 2025
Merged

Support full expression in WHERE clauses #3849

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c1f1062
Support full expression in WHERE clauses
LantaoJin Jul 4, 2025
9d92f8f
add unit tests
LantaoJin Jul 4, 2025
93ac6e1
Merge remote-tracking branch 'upstream/main' into issues/3273
LantaoJin Jul 4, 2025
4b2ef51
revert typo
LantaoJin Jul 4, 2025
5173edd
Fix IT
LantaoJin Jul 7, 2025
71001e0
Fix IT
LantaoJin Jul 7, 2025
2b8d0bd
Merge remote-tracking branch 'upstream/main' into issues/3273
LantaoJin Jul 18, 2025
8fe66a0
Address comment
LantaoJin Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion integ-test/src/test/java/org/opensearch/sql/ppl/WhereCommandIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ public void testWhereWithMultiLogicalExpr() throws IOException {
executeQuery(
String.format(
"source=%s "
+ "| where firstname='Amber' lastname='Duke' age=32 "
+ "| where firstname='Amber' and lastname='Duke' and age=32 "
+ "| fields firstname, lastname, age",
TEST_INDEX_ACCOUNT));
verifyDataRows(result, rows("Amber", "Duke", 32));
Expand Down
111 changes: 45 additions & 66 deletions ppl/src/main/antlr/OpenSearchPPLParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ pplStatement
;

dmlStatement
: queryStatement
| explainStatement
: explainStatement
| queryStatement
;

queryStatement
Expand All @@ -43,9 +43,9 @@ subSearch

// commands
pplCommands
: searchCommand
| describeCommand
: describeCommand
| showDataSourcesCommand
| searchCommand
;

commands
Expand Down Expand Up @@ -106,9 +106,9 @@ commandName
;

searchCommand
: (SEARCH)? fromClause # searchFrom
| (SEARCH)? fromClause logicalExpression # searchFromFilter
| (SEARCH)? logicalExpression fromClause # searchFilterFrom
: (SEARCH)? fromClause # searchFrom
| (SEARCH)? fromClause logicalExpression (logicalExpression)* # searchFromFilter
| SEARCH logicalExpression fromClause # searchFilterFrom

@LantaoJin LantaoJin Jul 8, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SEARCH keyword changed to required since the query describe source=t could be ambiguous.
describe source=t should throw syntax error "describe source= <==== .."
But it could be matched to

(SEARCH)? logicalExpression fromClause

logicalExpression -> valueExpr -> fieldExpr -> describe
fromClause -> source=t

@LantaoJin LantaoJin Jul 8, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why we supported the syntax of (SEARCH)? logicalExpression fromClause.

@penghuo penghuo Jul 16, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

search keywords is optional, this is valid query status=200 source=index

@LantaoJin LantaoJin Jul 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, now the logicalExpression could be both before and after source clause which is similar to SPL search command.

searchCommand
  : (SEARCH)? (logicalExpression)* fromClause (logicalExpression)*     # searchFrom

;

describeCommand
Expand Down Expand Up @@ -373,7 +373,7 @@ sortbyClause
;

evalClause
: fieldExpression EQUAL expression
: fieldExpression EQUAL logicalExpression
;

eventstatsAggTerm
Expand Down Expand Up @@ -447,68 +447,52 @@ numericLiteral
| floatLiteral
;

// expressions
expression
: logicalExpression
| comparisonExpression
| valueExpression
;

// predicates
logicalExpression
: LT_PRTHS logicalExpression RT_PRTHS # parentheticLogicalExpr
| NOT logicalExpression # logicalNot
| left = logicalExpression (AND)? right = logicalExpression # logicalAnd
: NOT logicalExpression # logicalNot
| left = logicalExpression AND right = logicalExpression # logicalAnd

@LantaoJin LantaoJin Jul 8, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AND will be required in Where expression. But it is still optional in Search filter expression:

source=t a=1 b=c

✅

source=t | where a=1 b=c

❌ (SPL cannot work either)

source=t | where a=1 and b=c

✅

This limitation is because functionArgs could be ambiguous. For example:

source=t | eval f=position('substr' IN 'str')

can be parsed to an incorrect syntax tree:
Screenshot 2025-07-08 at 16 39 31

The correct syntax tree is

Screenshot 2025-07-08 at 16 41 02

| left = logicalExpression XOR right = logicalExpression # logicalXor
| left = logicalExpression OR right = logicalExpression # logicalOr
| comparisonExpression # comparsion
| booleanExpression # booleanExpr
| relevanceExpression # relevanceExpr
| expression # logicalExpr
;

comparisonExpression
: left = valueExpression comparisonOperator right = valueExpression # compareExpr
| valueExpression NOT? IN valueList # inExpr
| valueExpression NOT? BETWEEN valueExpression AND valueExpression # between
;

valueExpressionList
: valueExpression
| LT_PRTHS valueExpression (COMMA valueExpression)* RT_PRTHS
expression
: valueExpression # valueExpr
| relevanceExpression # relevanceExpr
| left = expression comparisonOperator right = expression # compareExpr
| expression NOT? IN valueList # inExpr
| expression NOT? BETWEEN expression AND expression # between
;

valueExpression
: left = valueExpression binaryOperator = (STAR | DIVIDE | MODULE) right = valueExpression # binaryArithmetic
| left = valueExpression binaryOperator = (PLUS | MINUS) right = valueExpression # binaryArithmetic
| primaryExpression # valueExpressionDefault
| positionFunction # positionFunctionCall
| caseFunction # caseExpr
| extractFunction # extractFunctionCall
| getFormatFunction # getFormatFunctionCall
| timestampFunction # timestampFunctionCall
| LT_PRTHS valueExpression RT_PRTHS # parentheticValueExpr
| LT_SQR_PRTHS subSearch RT_SQR_PRTHS # scalarSubqueryExpr
| lambda # lambdaExpr
;

primaryExpression
: left = valueExpression binaryOperator = (STAR | DIVIDE | MODULE) right = valueExpression # binaryArithmetic
| left = valueExpression binaryOperator = (PLUS | MINUS) right = valueExpression # binaryArithmetic
| literalValue # literalValueExpr
| functionCall # functionCallExpr
| lambda # lambdaExpr
| LT_SQR_PRTHS subSearch RT_SQR_PRTHS # scalarSubqueryExpr
| valueExpression NOT? IN LT_SQR_PRTHS subSearch RT_SQR_PRTHS # inSubqueryExpr
| LT_PRTHS valueExpression (COMMA valueExpression)* RT_PRTHS NOT? IN LT_SQR_PRTHS subSearch RT_SQR_PRTHS # inSubqueryExpr
| EXISTS LT_SQR_PRTHS subSearch RT_SQR_PRTHS # existsSubqueryExpr
| fieldExpression # fieldExpr
| LT_PRTHS logicalExpression RT_PRTHS # nestedValueExpr
;

functionCall
: evalFunctionCall
| dataTypeFunctionCall
| fieldExpression
| literalValue
| positionFunctionCall
| caseFunctionCall
| timestampFunctionCall
| extractFunctionCall
| getFormatFunctionCall
;

positionFunction
positionFunctionCall
: positionFunctionName LT_PRTHS functionArg IN functionArg RT_PRTHS
;

booleanExpression
: booleanFunctionCall # booleanFunctionCallExpr
| valueExpressionList NOT? IN LT_SQR_PRTHS subSearch RT_SQR_PRTHS # inSubqueryExpr
| EXISTS LT_SQR_PRTHS subSearch RT_SQR_PRTHS # existsSubqueryExpr
;

caseFunction
caseFunctionCall
: CASE LT_PRTHS logicalExpression COMMA valueExpression (COMMA logicalExpression COMMA valueExpression)* (ELSE valueExpression)? RT_PRTHS
;

Expand Down Expand Up @@ -573,12 +557,7 @@ evalFunctionCall

// cast function
dataTypeFunctionCall
: CAST LT_PRTHS expression AS convertedDataType RT_PRTHS
;

// boolean functions
booleanFunctionCall
: conditionFunctionName LT_PRTHS functionArgs RT_PRTHS
: CAST LT_PRTHS logicalExpression AS convertedDataType RT_PRTHS
;

convertedDataType
Expand Down Expand Up @@ -621,12 +600,12 @@ functionArg

functionArgExpression
: lambda
| expression
| logicalExpression
;

lambda
: ident ARROW expression
| LT_PRTHS ident (COMMA ident)+ RT_PRTHS ARROW expression
: ident ARROW logicalExpression
| LT_PRTHS ident (COMMA ident)+ RT_PRTHS ARROW logicalExpression
;

relevanceArg
Expand Down Expand Up @@ -837,7 +816,7 @@ dateTimeFunctionName
| YEARWEEK
;

getFormatFunction
getFormatFunctionCall
: GET_FORMAT LT_PRTHS getFormatType COMMA functionArg RT_PRTHS
;

Expand All @@ -848,7 +827,7 @@ getFormatType
| TIMESTAMP
;

extractFunction
extractFunctionCall
: EXTRACT LT_PRTHS datetimePart FROM functionArg RT_PRTHS
;

Expand Down Expand Up @@ -883,7 +862,7 @@ datetimePart
| complexDateTimePart
;

timestampFunction
timestampFunctionCall
: timestampFunctionName LT_PRTHS simpleDateTimePart COMMA firstArg = functionArg COMMA secondArg = functionArg RT_PRTHS
;

Expand Down
7 changes: 6 additions & 1 deletion ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import org.opensearch.sql.ast.dsl.AstDSL;
import org.opensearch.sql.ast.expression.Alias;
import org.opensearch.sql.ast.expression.AllFieldsExcludeMeta;
import org.opensearch.sql.ast.expression.And;
import org.opensearch.sql.ast.expression.EqualTo;
import org.opensearch.sql.ast.expression.Field;
import org.opensearch.sql.ast.expression.Let;
Expand Down Expand Up @@ -144,7 +145,11 @@ public UnresolvedPlan visitSearchFrom(SearchFromContext ctx) {

@Override
public UnresolvedPlan visitSearchFromFilter(SearchFromFilterContext ctx) {
return new Filter(internalVisitExpression(ctx.logicalExpression()))
return new Filter(
ctx.logicalExpression().stream()
.map(this::internalVisitExpression)
.reduce(And::new)
.get())
.attach(visit(ctx.fromClause()));
}

Expand Down
Loading
Loading