KE-35827 new second storage sql pushdown#453

Merged

chenzhx merged 53 commits intokyspark-3.2.x-4.xfrom

kyspark-3.2.x-4.x-agg-pushdown

May 5, 2022

Commits on Apr 18, 2022

[SPARK-36556][SQL] Add DSV2 filters

authored and
chenzhx
committed
[SPARK-36760][SQL] Add interface SupportsPushDownV2Filters

authored and
chenzhx
committed
[SPARK-37020][SQL] DS V2 LIMIT push down

huaxingao
authored and
chenzhx
committed
[SPARK-37038][SQL] DSV2 Sample Push Down

huaxingao
authored and
chenzhx
committed
[SPARK-37286][SQL] Move compileAggregates from JDBCRDD to JdbcDialect

beliefer
authored and
chenzhx
committed
[SPARK-37286][DOCS][FOLLOWUP] Fix the wrong parameter name for Javadoc

sarutak
authored and
chenzhx
committed
[SPARK-37262][SQL] Don't log empty aggregate and group by in JDBCScan

huaxingao
authored and
chenzhx
committed
[SPARK-37483][SQL] Support push down top N to JDBC data source V2

beliefer
authored and
chenzhx
committed
[SPARK-37644][SQL] Support datasource v2 complete aggregate pushdown

beliefer
authored and
chenzhx
committed
[SPARK-37627][SQL] Add sorted column in BucketTransform

huaxingao
authored and
chenzhx
committed
[SPARK-37789][SQL] Add a class to represent general aggregate functions in DS V2

cloud-fan
authored and
chenzhx
committed
[SPARK-37644][SQL][FOLLOWUP] When partition column is same as group by key, pushing down aggregate completely

beliefer
authored and
chenzhx
committed
[SPARK-37527][SQL] Translate more standard aggregate functions for pushdown

beliefer
authored and
chenzhx
committed
[SPARK-37734][SQL][TESTS] Upgrade h2 from 1.4.195 to 2.0.204

beliefer
authored and
chenzhx
committed
[SPARK-37527][SQL] Compile `COVAR_POP`, `COVAR_SAMP` and `CORR` in `H2Dialet`

beliefer
authored and
chenzhx
committed
[SPARK-37839][SQL] DS V2 supports partial aggregate push-down `AVG`

beliefer
authored and
chenzhx
committed
[SPARK-36526][SQL] DSV2 Index Support: Add supportsIndex interface

huaxingao
authored and
chenzhx
committed
[SPARK-36913][SQL] Implement createIndex and IndexExists in DS V2 JDBC (MySQL dialect)

huaxingao
authored and
chenzhx
committed
[SPARK-36914][SQL] Implement dropIndex and listIndexes in JDBC (MySQL dialect)

huaxingao
authored and
chenzhx
committed
[SPARK-37343][SQL] Implement createIndex, IndexExists and dropIndex in JDBC (Postgres dialect)

dchvn
authored and
chenzhx
committed
[SPARK-37867][SQL] Compile aggregate functions of build-in JDBC dialect

beliefer
authored and
chenzhx
committed
[SPARK-37929][SQL][FOLLOWUP] Support cascade mode for JDBC V2

beliefer
authored and
chenzhx
committed
[SPARK-38035][SQL] Add docker tests for build-in JDBC dialect

beliefer
authored and
chenzhx
committed
[SPARK-38054][SQL] Supports list namespaces in JDBC v2 MySQL dialect

beliefer
authored and
chenzhx
committed
[SPARK-36351][SQL] Refactor filter push down in file source v2

huaxingao
authored and
chenzhx
committed
[SPARK-36645][SQL] Aggregate (Min/Max/Count) push down for Parquet

huaxingao
authored and
chenzhx
committed
[SPARK-34960][SQL] Aggregate push down for ORC

c21
authored and
chenzhx
committed
[SPARK-37960][SQL] A new framework to represent catalyst expressions in DS v2 APIs

beliefer
authored and
chenzhx
committed
[SPARK-37867][SQL][FOLLOWUP] Compile aggregate functions for build-in DB2 dialect

beliefer
authored and
chenzhx
committed
[SPARK-36568][SQL] Better FileScan statistics estimation

peter-toth
authored and
chenzhx
committed
[SPARK-37929][SQL] Support cascade mode for `dropNamespace` API

dchvn
authored and
chenzhx
committed
code format
chenzhx
committed
[SPARK-38196][SQL] Refactor framework so as JDBC dialect could compile expression by self way

beliefer
authored and
chenzhx
committed
[SPARK-38361][SQL] Add factory method `getConnection` into `JDBCDialect`

beliefer
authored and
chenzhx
committed
code format
chenzhx
committed
[SPARK-38560][SQL] If `Sum`, `Count`, `Any` accompany with distinct, cannot do partial agg push down

beliefer
authored and
chenzhx
committed
[SPARK-36718][SQL] Only collapse projects if we don't duplicate expensive expressions

cloud-fan
authored and
chenzhx
committed
[SPARK-38432][SQL] Refactor framework so as JDBC dialect could compile filter by self way

beliefer
authored and
chenzhx
committed
[SPARK-38432][SQL][FOLLOWUP] Supplement test case for overflow and add comments

beliefer
authored and
chenzhx
committed
[SPARK-38533][SQL] DS V2 aggregate push-down supports project with alias

beliefer
authored and
chenzhx
committed
code foramt
chenzhx
committed
[SPARK-37483][SQL][FOLLOWUP] Rename `pushedTopN` to `PushedTopN` and improve JDBCV2Suite

beliefer
authored and
chenzhx
committed
[SPARK-38644][SQL] DS V2 topN push-down supports project with alias

beliefer
authored and
chenzhx
committed
[SPARK-38391][SQL] Datasource v2 supports partial topN push-down

beliefer
authored and
chenzhx
committed
[SPARK-38633][SQL] Support push down Cast to JDBC data source V2

beliefer
authored and
chenzhx
committed
[SPARK-38432][SQL][FOLLOWUP] Add test case for push down filter with alias

beliefer
authored and
chenzhx
committed
[SPARK-38633][SQL][FOLLOWUP] JDBCSQLBuilder should build cast to type of databases

beliefer
authored and
chenzhx
committed
[SPARK-37839][SQL][FOLLOWUP] Check overflow when DS V2 partial aggregate push-down `AVG`

beliefer
authored and
chenzhx
committed
[SPARK-37960][SQL][FOLLOWUP] Make the testing CASE WHEN query more reasonable

beliefer
authored and
chenzhx
committed
[SPARK-38761][SQL] DS V2 supports push down misc non-aggregate functions

beliefer
authored and
chenzhx
committed
[SPARK-38865][SQL][DOCS] Update document of JDBC options for `pushDownAggregate` and `pushDownLimit`

beliefer
authored and
chenzhx
committed
[SPARK-38855][SQL] DS V2 supports push down math functions

beliefer
authored and
chenzhx
committed

Commits on Apr 29, 2022

update spark version to r61
chenzhx
committed