Skip to content

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 6 commits into from
Closed

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3dc0124
Reduce shuffle partitions to reduce testing time
wangyum Sep 22, 2019
6ec9761
Avoid test error in StreamSuite
wangyum Sep 22, 2019
55004b9
SHUFFLE_PARTITIONS to 4
wangyum Sep 22, 2019
b4f2d19
SHUFFLE_PARTITIONS to 5
wangyum Sep 23, 2019
ad6bee7
SHUFFLE_PARTITIONS to 4 and add sort
wangyum Sep 23, 2019
df51b69
DESC -> ASC
wangyum Sep 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-joins.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ GROUP BY t1a,
t3a,
t3b,
t3c
ORDER BY t1a DESC, t3b DESC;
ORDER BY t1a DESC, t3b DESC, t3c ASC;

-- TC 01.03
SELECT Count(DISTINCT(t1a))
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ GROUP BY t1b,
HAVING t1d NOT IN (SELECT t2d
FROM t2
WHERE t1d = t2d)
ORDER BY t1b DESC;
ORDER BY t1b DESC, t1d ASC;

-- TC 01.05
SELECT COUNT(DISTINCT(t1a)),
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ GROUP BY t1a,
t3a,
t3b,
t3c
ORDER BY t1a DESC, t3b DESC
ORDER BY t1a DESC, t3b DESC, t3c ASC
-- !query 4 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t3a:string,t3b:smallint,t3c:int>
-- !query 4 output
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ GROUP BY t1b,
HAVING t1d NOT IN (SELECT t2d
FROM t2
WHERE t1d = t2d)
ORDER BY t1b DESC
ORDER BY t1b DESC, t1d ASC
-- !query 6 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t1c:int,t1d:bigint>
-- !query 6 output
Expand Down
10 changes: 5 additions & 5 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.util.{Locale, TimeZone}

import scala.util.control.NonFatal

import org.apache.spark.SparkException
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.plans.logical.sql.{DescribeColumnStatement, DescribeTableStatement}
Expand Down Expand Up @@ -133,6 +133,10 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
private val notIncludedMsg = "[not included in comparison]"
private val clsName = this.getClass.getCanonicalName

override def sparkConf: SparkConf = super.sparkConf
// Fewer shuffle partitions to speed up testing.
.set(SQLConf.SHUFFLE_PARTITIONS, 4)

/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
"blacklist.sql" // Do NOT remove this one. It is here to test the blacklist functionality.
Expand Down Expand Up @@ -291,10 +295,6 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {

testCase match {
case udfTestCase: UDFTest =>
// In Python UDF tests, the number of shuffle partitions matters considerably in
// the testing time because it requires to fork and communicate between external
// processes.
localSparkSession.conf.set(SQLConf.SHUFFLE_PARTITIONS.key, 4)
registerTestUDF(udfTestCase.udf, localSparkSession)
case _ =>
}
Expand Down