New trendline ppl command (SMA only)#833
Conversation
YANG-DB
left a comment
There was a problem hiding this comment.
Please also add command to the README, a dedicated page for this command with a link to the dedicated doc and examples in the examples page
2c04f75 to
4a5d3c1
Compare
There was a problem hiding this comment.
@salyh plz add logical plan test in the relevant folder
And also plz add a test with no alias example (for logical and actual queries)
done |
|
@jduo thanks for u'r feedback - really appreciated ! |
There was a problem hiding this comment.
To save time, you can ignore the plan assertion and just check the return data matching the expectation. Many thanks! Here is an example for chaining trendline:
| source = $testTable | eval age_1 = age, age_2 = age | trendline sort - age_1 sma(3, age_1) | trendline sort + age_2 sma(3, age_2)
|
Another question is whether the sort field can be different from the calculated field? If the answer is no, please throw out an exception for But this However, I can give an approval to unblock releasing this as is for now as long as above chaining case works. |
@LantaoJin @YANG-DB IT chaining test added |
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
5a51ceb to
ffcc918
Compare
@LantaoJin I'm approving the PR for merge as this issue was addresses here |
* update antlr grammar for (future) P1 command syntax Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add trendline command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add expand command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add geoip command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * PPl `flatten` command (#784) * The flatten command implemented Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * The flatten command integration tests were extended with additional checks for logical plans. Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * flatten, added more tests related to plan translation and integration tests Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * Flatten command added to command names list. Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> --------- Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * Extract source table names from mv query (#854) * add sourceTables to MV index metadata properties Signed-off-by: Sean Kao <seankao@amazon.com> * parse source tables from mv query Signed-off-by: Sean Kao <seankao@amazon.com> * test cases for parse source tables from mv query Signed-off-by: Sean Kao <seankao@amazon.com> * use constant for metadata cache version Signed-off-by: Sean Kao <seankao@amazon.com> * write source tables to metadata cache Signed-off-by: Sean Kao <seankao@amazon.com> * address comment Signed-off-by: Sean Kao <seankao@amazon.com> * generate source tables for old mv without new prop Signed-off-by: Sean Kao <seankao@amazon.com> * syntax fix Signed-off-by: Sean Kao <seankao@amazon.com> --------- Signed-off-by: Sean Kao <seankao@amazon.com> * Fallback to internal scheduler when index creation failed (#850) * Fallback to internal scheduler when index creation failed Signed-off-by: Louis Chu <clingzhi@amazon.com> * Fix IT Signed-off-by: Louis Chu <clingzhi@amazon.com> * Fix IOException Signed-off-by: Louis Chu <clingzhi@amazon.com> --------- Signed-off-by: Louis Chu <clingzhi@amazon.com> * New trendline ppl command (SMA only) (#833) * WIP trendline command Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * wip Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * trendline supports sorting Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * run scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * return null when there are too few data points Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * sbt scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Remove WMA references Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * trendline - sortByField as Optional<Field> Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * introduce TrendlineStrategy Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * keywordsCanBeId -> replace SMA with trendlineType Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * handle trendline alias as qualifiedName instead of fieldExpression Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Add docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Make alias optional Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adapt tests for optional alias Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adden logical plan unittests Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add missing license headers Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * numberOfDataPoints must be 1 or greater Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Rename TrendlineStrategy to TrendlineCatalystUtils Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Validate TrendlineType early and pass around enum type Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add trendline chaining test Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix compile errors Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> --------- Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * update iplocation antlr Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> Signed-off-by: Sean Kao <seankao@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: lukasz-soszynski-eliatra <110241464+lukasz-soszynski-eliatra@users.noreply.github.com> Co-authored-by: Sean Kao <seankao@amazon.com> Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
* update antlr grammar for (future) P1 command syntax Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add trendline command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add expand command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add geoip command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * PPl `flatten` command (#784) * The flatten command implemented Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * The flatten command integration tests were extended with additional checks for logical plans. Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * flatten, added more tests related to plan translation and integration tests Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * Flatten command added to command names list. Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> --------- Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> * Extract source table names from mv query (#854) * add sourceTables to MV index metadata properties Signed-off-by: Sean Kao <seankao@amazon.com> * parse source tables from mv query Signed-off-by: Sean Kao <seankao@amazon.com> * test cases for parse source tables from mv query Signed-off-by: Sean Kao <seankao@amazon.com> * use constant for metadata cache version Signed-off-by: Sean Kao <seankao@amazon.com> * write source tables to metadata cache Signed-off-by: Sean Kao <seankao@amazon.com> * address comment Signed-off-by: Sean Kao <seankao@amazon.com> * generate source tables for old mv without new prop Signed-off-by: Sean Kao <seankao@amazon.com> * syntax fix Signed-off-by: Sean Kao <seankao@amazon.com> --------- Signed-off-by: Sean Kao <seankao@amazon.com> * Fallback to internal scheduler when index creation failed (#850) * Fallback to internal scheduler when index creation failed Signed-off-by: Louis Chu <clingzhi@amazon.com> * Fix IT Signed-off-by: Louis Chu <clingzhi@amazon.com> * Fix IOException Signed-off-by: Louis Chu <clingzhi@amazon.com> --------- Signed-off-by: Louis Chu <clingzhi@amazon.com> * New trendline ppl command (SMA only) (#833) * WIP trendline command Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * wip Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * trendline supports sorting Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * run scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * return null when there are too few data points Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * sbt scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Remove WMA references Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * trendline - sortByField as Optional<Field> Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * introduce TrendlineStrategy Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * keywordsCanBeId -> replace SMA with trendlineType Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * handle trendline alias as qualifiedName instead of fieldExpression Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Add docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Make alias optional Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adapt tests for optional alias Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adden logical plan unittests Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add missing license headers Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * numberOfDataPoints must be 1 or greater Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Rename TrendlineStrategy to TrendlineCatalystUtils Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Validate TrendlineType early and pass around enum type Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add trendline chaining test Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix compile errors Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> --------- Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * update iplocation antlr Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala fmt style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * `cidrmatch` ppl command add logical tests and docs (#865) * update logical tests and docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala fmt style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix type error Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> * Support Lambda and add related array functions (#864) * json function enhancement Signed-off-by: Heng Qian <qianheng@amazon.com> * Add JavaToScalaTransformer Signed-off-by: Heng Qian <qianheng@amazon.com> * Apply scalafmtAll Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Add IT and change to use the same function name as spark Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Add document and separate lambda functions from json functions Signed-off-by: Heng Qian <qianheng@amazon.com> * Add lambda functions transform and reduce Signed-off-by: Heng Qian <qianheng@amazon.com> * polish lambda function document Signed-off-by: Heng Qian <qianheng@amazon.com> * polish lambda function document Signed-off-by: Heng Qian <qianheng@amazon.com> * Minor fix Signed-off-by: Heng Qian <qianheng@amazon.com> * Minor change to polish the documents Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com> Signed-off-by: Sean Kao <seankao@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> Signed-off-by: Heng Qian <qianheng@amazon.com> Co-authored-by: lukasz-soszynski-eliatra <110241464+lukasz-soszynski-eliatra@users.noreply.github.com> Co-authored-by: Sean Kao <seankao@amazon.com> Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Co-authored-by: qianheng <qianheng@amazon.com>
* WIP trendline command Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * wip Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * trendline supports sorting Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * run scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * return null when there are too few data points Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * sbt scalafmtAll Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Remove WMA references Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * trendline - sortByField as Optional<Field> Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * introduce TrendlineStrategy Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * keywordsCanBeId -> replace SMA with trendlineType Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * handle trendline alias as qualifiedName instead of fieldExpression Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> * Add docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Make alias optional Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adapt tests for optional alias Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Adden logical plan unittests Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add missing license headers Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix docs Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * numberOfDataPoints must be 1 or greater Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Rename TrendlineStrategy to TrendlineCatalystUtils Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Validate TrendlineType early and pass around enum type Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Add trendline chaining test Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix compile errors Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> * Fix imports Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> --------- Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com> Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com> Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Description
New trendline ppl command (SMA only). For WMA see #831
Related Issues
#655
Check List
--signoffBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.