Skip to content

P1 Antlr grammar#859

Merged
YANG-DB merged 10 commits intoopensearch-project:p1-antlr-grammarfrom
YANG-DB:p1-antlr-grammar
Nov 2, 2024
Merged

P1 Antlr grammar#859
YANG-DB merged 10 commits intoopensearch-project:p1-antlr-grammarfrom
YANG-DB:p1-antlr-grammar

Conversation

@YANG-DB
Copy link
Copy Markdown
Member

@YANG-DB YANG-DB commented Oct 31, 2024

Description

Add latest p1 commands into the Antlr grammar:

  • flatten
  • geoip
  • trendline
  • expand

Check List

  • Updated documentation (docs/ppl-lang/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • New added source code should include a copyright header
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@YANG-DB
update antlr grammar for (future) P1 command syntax
ff222c9 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add trendline command
281c641 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add expand command
596c069 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add geoip command
4439f06 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
ykmr1224
ykmr1224 reviewed Oct 31, 2024
View reviewed changes
Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
Comment on lines +257 to +258
: FLATTEN fieldExpression
;
Copy link
Copy Markdown
Collaborator

@ykmr1224 ykmr1224 Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is the indent level correct? (save for below)

Copy link
Copy Markdown
Member Author

@YANG-DB YANG-DB Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it is the same with the following
kmeansCommand, adCommand and so on...

Copy link
Copy Markdown
Member

@LantaoJin LantaoJin Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no right or wrong about 3 spaces or 4 spaces as indent level. All indent level should be same.

Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
;

expandCommand
: EXPAND fieldExpression
Copy link
Copy Markdown
Collaborator

@ykmr1224 ykmr1224 Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we use expand instead of expand_field? (ref)

Copy link
Copy Markdown
Member Author

@YANG-DB YANG-DB Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

@lukasz-soszynski-eliatra
PPl flatten command (opensearch-project#784)
b71a9ed 
* The flatten command implemented

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* The flatten command integration tests were extended with additional checks for logical plans.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* flatten, added more tests related to plan translation and integration tests

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* Flatten command added to command names list.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

---------

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>
@YANG-DB YANG-DB requested a review from ykmr1224 October 31, 2024 22:39
ykmr1224
ykmr1224 reviewed Oct 31, 2024
View reviewed changes
Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
BETWEEN: 'BETWEEN';
CIDRMATCH: 'CIDRMATCH';

// Geo Loction
Copy link
Copy Markdown
Collaborator

@ykmr1224 ykmr1224 Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: Loction -> Location

@seankao31
Extract source table names from mv query (opensearch-project#854)
950009b 
* add sourceTables to MV index metadata properties

Signed-off-by: Sean Kao <seankao@amazon.com>

* parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* test cases for parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* use constant for metadata cache version

Signed-off-by: Sean Kao <seankao@amazon.com>

* write source tables to metadata cache

Signed-off-by: Sean Kao <seankao@amazon.com>

* address comment

Signed-off-by: Sean Kao <seankao@amazon.com>

* generate source tables for old mv without new prop

Signed-off-by: Sean Kao <seankao@amazon.com>

* syntax fix

Signed-off-by: Sean Kao <seankao@amazon.com>

---------

Signed-off-by: Sean Kao <seankao@amazon.com>
LantaoJin
LantaoJin reviewed Nov 1, 2024
View reviewed changes
Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4 Outdated
| FILLNULL
| FIELDSUMMARY
| TRENDLINE
| GEOIP
Copy link
Copy Markdown
Member

@LantaoJin LantaoJin Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GETOIP is not a command name

Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
Comment on lines +257 to +258
: FLATTEN fieldExpression
;
Copy link
Copy Markdown
Member

@LantaoJin LantaoJin Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no right or wrong about 3 spaces or 4 spaces as indent level. All indent level should be same.

Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4

// geoip function
geoipFunctionCall
: GEOIP LT_PRTHS (datasource = functionArg COMMA)? ipAddress = functionArg (COMMA properties = stringLiteral)? RT_PRTHS
Copy link
Copy Markdown
Member

@LantaoJin LantaoJin Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking why not reusing evalFunctionCall? I don't see any special syntax definition here. And IP function sounds not one of the primary expressions.

noCharger and others added 4 commits November 1, 2024 06:23
@noCharger
Fallback to internal scheduler when index creation failed (opensearch…
faae818 
…-project#850)

* Fallback to internal scheduler when index creation failed

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IT

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IOException

Signed-off-by: Louis Chu <clingzhi@amazon.com>

---------

Signed-off-by: Louis Chu <clingzhi@amazon.com>
@salyh @kt-eliatra
New trendline ppl command (SMA only) (opensearch-project#833)
bdb4848 
* WIP trendline command

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* wip

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* trendline supports sorting

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* run scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* return null when there are too few data points

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* sbt scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Remove WMA references

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* trendline - sortByField as Optional<Field>

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* introduce TrendlineStrategy

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* keywordsCanBeId -> replace SMA with trendlineType

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* handle trendline alias as qualifiedName instead of fieldExpression

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Add docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Make alias optional

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adapt tests for optional alias

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adden logical plan unittests

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add missing license headers

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* numberOfDataPoints must be 1 or greater

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Rename TrendlineStrategy to  TrendlineCatalystUtils

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Validate TrendlineType early and pass around enum type

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add trendline chaining test

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix compile errors

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

---------

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
@YANG-DB
update iplocation antlr
8f3405b 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
Merge branch 'main' into p1-antlr-grammar
c2acd9b 
# Conflicts:
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
@YANG-DB YANG-DB merged commit 0e733f5 into opensearch-project:p1-antlr-grammar Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LantaoJin LantaoJin LantaoJin left review comments

@ykmr1224 ykmr1224 ykmr1224 approved these changes

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@kaituo kaituo Awaiting requested review from kaituo kaituo is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@YANG-DB @LantaoJin @ykmr1224 @lukasz-soszynski-eliatra @seankao31 @noCharger @salyh