From 4e2861c19461c6381e89325738cb28bcac60f58b Mon Sep 17 00:00:00 2001 From: Jiaan Geng Date: Tue, 12 Apr 2022 09:29:30 +0800 Subject: [PATCH 1/4] [SPARK-38865][SQL][DOCS] Update document of JDBC options for pushDownAggregate and pushDownLimit --- docs/sql-data-sources-jdbc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/sql-data-sources-jdbc.md b/docs/sql-data-sources-jdbc.md index c57bc072df9e2..68872a50e6630 100644 --- a/docs/sql-data-sources-jdbc.md +++ b/docs/sql-data-sources-jdbc.md @@ -241,7 +241,7 @@ logging into the data sources. pushDownAggregate false - The option to enable or disable aggregate push-down in V2 JDBC data source. The default value is false, in which case Spark will not push down aggregates to the JDBC data source. Otherwise, if sets to true, aggregates will be pushed down to the JDBC data source. Aggregate push-down is usually turned off when the aggregate is performed faster by Spark than by the JDBC data source. Please note that aggregates can be pushed down if and only if all the aggregate functions and the related filters can be pushed down. Spark assumes that the data source can't fully complete the aggregate and does a final aggregate over the data source output. + The option to enable or disable aggregate push-down in V2 JDBC data source. The default value is false, in which case Spark will not push down aggregates to the JDBC data source. Otherwise, if sets to true, aggregates will be pushed down to the JDBC data source. Aggregate push-down is usually turned off when the aggregate is performed faster by Spark than by the JDBC data source. Please note that aggregates can be pushed down if and only if all the aggregate functions and the related filters can be pushed down. If numPartitions equals to 1 or the group by key is the same as partitionColumn, Spark will push down aggregate to data source completely and not apply a final aggregate over the data source output. Otherwise, Spark will apply a final aggregate over the data source output. read @@ -250,7 +250,7 @@ logging into the data sources. pushDownLimit false - The option to enable or disable LIMIT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT to the JDBC data source. Otherwise, if value sets to true, LIMIT is pushed down to the JDBC data source. SPARK still applies LIMIT on the result from data source even if LIMIT is pushed down. + The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. read From 55fe7462b349166fe7c3b3f5b98a88422efd0759 Mon Sep 17 00:00:00 2001 From: Jiaan Geng Date: Tue, 12 Apr 2022 09:42:36 +0800 Subject: [PATCH 2/4] Update code --- docs/sql-data-sources-jdbc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-data-sources-jdbc.md b/docs/sql-data-sources-jdbc.md index 68872a50e6630..6154eb7c32d70 100644 --- a/docs/sql-data-sources-jdbc.md +++ b/docs/sql-data-sources-jdbc.md @@ -250,7 +250,7 @@ logging into the data sources. pushDownLimit false - The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. + The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. read From a5b4c0e95cec9ff77439b77be1c3f91cf7b1c5de Mon Sep 17 00:00:00 2001 From: Jiaan Geng Date: Tue, 12 Apr 2022 09:44:35 +0800 Subject: [PATCH 3/4] Update code --- docs/sql-data-sources-jdbc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-data-sources-jdbc.md b/docs/sql-data-sources-jdbc.md index 6154eb7c32d70..68872a50e6630 100644 --- a/docs/sql-data-sources-jdbc.md +++ b/docs/sql-data-sources-jdbc.md @@ -250,7 +250,7 @@ logging into the data sources. pushDownLimit false - The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. + The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. read From 0987dc40aea4720296427678667366616a3f81e3 Mon Sep 17 00:00:00 2001 From: Jiaan Geng Date: Tue, 12 Apr 2022 19:35:22 +0800 Subject: [PATCH 4/4] Update code --- docs/sql-data-sources-jdbc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-data-sources-jdbc.md b/docs/sql-data-sources-jdbc.md index 68872a50e6630..e17c8f686fce5 100644 --- a/docs/sql-data-sources-jdbc.md +++ b/docs/sql-data-sources-jdbc.md @@ -250,7 +250,7 @@ logging into the data sources. pushDownLimit false - The option to enable or disable LIMIT push-down into V2 JDBC data source. If LIMIT with SORT, the option to enable or disable LIMIT with SORT push-down into V2 JDBC data source. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. + The option to enable or disable LIMIT push-down into V2 JDBC data source. The LIMIT push-down also includes LIMIT + SORT , a.k.a. the Top N operator. The default value is false, in which case Spark does not push down LIMIT or LIMIT with SORT to the JDBC data source. Otherwise, if sets to true, LIMIT or LIMIT with SORT is pushed down to the JDBC data source. If numPartitions is greater than 1, SPARK still applies LIMIT or LIMIT with SORT on the result from data source even if LIMIT or LIMIT with SORT is pushed down. Otherwise, if LIMIT or LIMIT with SORT is pushed down and numPartitions equals to 1, SPARK will not apply LIMIT or LIMIT with SORT on the result from data source. read