Skip to content

P1 antlr grammar - update lambda & join #874

Merged
YANG-DB merged 15 commits intoopensearch-project:p1-antlr-grammarfrom
YANG-DB:p1-antlr-grammar
Nov 5, 2024
Merged

P1 antlr grammar - update lambda & join #874
YANG-DB merged 15 commits intoopensearch-project:p1-antlr-grammarfrom
YANG-DB:p1-antlr-grammar

Conversation

@YANG-DB
Copy link
Copy Markdown
Member

@YANG-DB YANG-DB commented Nov 5, 2024

Description

update ANTLR with lambda & join grammar changes

Check List

  • Updated documentation (docs/ppl-lang/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • New added source code should include a copyright header
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

YANG-DB and others added 15 commits October 31, 2024 11:52
@YANG-DB
update antlr grammar for (future) P1 command syntax
ff222c9 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add trendline command
281c641 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add expand command
596c069 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
add geoip command
4439f06 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@lukasz-soszynski-eliatra
PPl flatten command (opensearch-project#784)
b71a9ed 
* The flatten command implemented

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* The flatten command integration tests were extended with additional checks for logical plans.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* flatten, added more tests related to plan translation and integration tests

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* Flatten command added to command names list.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

---------

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>
@seankao31
Extract source table names from mv query (opensearch-project#854)
950009b 
* add sourceTables to MV index metadata properties

Signed-off-by: Sean Kao <seankao@amazon.com>

* parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* test cases for parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* use constant for metadata cache version

Signed-off-by: Sean Kao <seankao@amazon.com>

* write source tables to metadata cache

Signed-off-by: Sean Kao <seankao@amazon.com>

* address comment

Signed-off-by: Sean Kao <seankao@amazon.com>

* generate source tables for old mv without new prop

Signed-off-by: Sean Kao <seankao@amazon.com>

* syntax fix

Signed-off-by: Sean Kao <seankao@amazon.com>

---------

Signed-off-by: Sean Kao <seankao@amazon.com>
@noCharger
Fallback to internal scheduler when index creation failed (opensearch…
faae818 
…-project#850)

* Fallback to internal scheduler when index creation failed

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IT

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IOException

Signed-off-by: Louis Chu <clingzhi@amazon.com>

---------

Signed-off-by: Louis Chu <clingzhi@amazon.com>
@salyh @kt-eliatra
New trendline ppl command (SMA only) (opensearch-project#833)
bdb4848 
* WIP trendline command

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* wip

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* trendline supports sorting

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* run scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* return null when there are too few data points

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* sbt scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Remove WMA references

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* trendline - sortByField as Optional<Field>

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* introduce TrendlineStrategy

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* keywordsCanBeId -> replace SMA with trendlineType

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* handle trendline alias as qualifiedName instead of fieldExpression

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Add docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Make alias optional

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adapt tests for optional alias

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adden logical plan unittests

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add missing license headers

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* numberOfDataPoints must be 1 or greater

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Rename TrendlineStrategy to  TrendlineCatalystUtils

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Validate TrendlineType early and pass around enum type

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add trendline chaining test

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix compile errors

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

---------

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
@YANG-DB
update iplocation antlr
8f3405b 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
Merge branch 'main' into p1-antlr-grammar
c2acd9b 
# Conflicts:
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
@YANG-DB
update scala fmt style
6b44cb3 
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@YANG-DB
cidrmatch ppl command add logical tests and docs (opensearch-projec…
b183b4a 
…t#865)

* update logical tests and docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update scala fmt style

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix type error

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
@qianheng-aws
Support Lambda and add related array functions (opensearch-project#864)
aaba489 
* json function enhancement

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add JavaToScalaTransformer

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Apply scalafmtAll

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add IT and change to use the same function name as spark

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add document and separate lambda functions from json functions

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add lambda functions transform and reduce

Signed-off-by: Heng Qian <qianheng@amazon.com>

* polish lambda function document

Signed-off-by: Heng Qian <qianheng@amazon.com>

* polish lambda function document

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Minor fix

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Minor change to polish the documents

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@YANG-DB
Merge branch 'main' into p1-antlr-grammar
f5f05f8 
# Conflicts:
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
@YANG-DB
Merge remote-tracking branch 'origin/p1-antlr-grammar' into p1-antlr-…
bd13a64 
…grammar

# Conflicts:
#	docs/ppl-lang/PPL-Example-Commands.md
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystExpressionVisitor.java
@YANG-DB YANG-DB added the Lang:PPL Pipe Processing Language support label Nov 5, 2024
ykmr1224
ykmr1224 approved these changes Nov 5, 2024
View reviewed changes
Comment thread ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
ARRAY: 'ARRAY';

// LAMBDA FUNCTIONS
//EXISTS: 'EXISTS';
Copy link
Copy Markdown
Collaborator

@ykmr1224 ykmr1224 Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this commented out?

Copy link
Copy Markdown
Member Author

@YANG-DB YANG-DB Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not supported yet by the lambda command

Copy link
Copy Markdown
Member

@LantaoJin LantaoJin Nov 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented EXISTS out because the it is an existed keyword. Just add it here as a placeholder. I think this line can be deleted to reduce confusion.

@YANG-DB YANG-DB merged commit f8c7c80 into opensearch-project:p1-antlr-grammar Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ykmr1224 ykmr1224 ykmr1224 approved these changes

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@kaituo kaituo Awaiting requested review from kaituo kaituo is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

Assignees

No one assigned

Labels

Lang:PPL Pipe Processing Language support

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@YANG-DB @LantaoJin @ykmr1224 @lukasz-soszynski-eliatra @seankao31 @noCharger @salyh @qianheng-aws