Skip to content

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 6 commits into from
Closed

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3dc0124
Reduce shuffle partitions to reduce testing time
wangyum Sep 22, 2019
6ec9761
Avoid test error in StreamSuite
wangyum Sep 22, 2019
55004b9
SHUFFLE_PARTITIONS to 4
wangyum Sep 22, 2019
b4f2d19
SHUFFLE_PARTITIONS to 5
wangyum Sep 23, 2019
ad6bee7
SHUFFLE_PARTITIONS to 4 and add sort
wangyum Sep 23, 2019
df51b69
DESC -> ASC
wangyum Sep 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ ORDER BY t1a DESC, t3b DESC
-- !query 4 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t3a:string,t3b:smallint,t3c:int>
-- !query 4 output
1 10 val3b 8 NULL
1 10 val1b 8 16
1 10 val3b 8 NULL

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, do we really need this?

@wangyum wangyum Sep 22, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

[info] - subquery/in-subquery/in-joins.sql *** FAILED *** (15 seconds, 359 milliseconds)
[info]   subquery/in-subquery/in-joins.sql
[info]   Expected "1    10      val[3b  8       NULL
[info]   1      10      val1b   8       16]
[info]   1      10      val3a   6       12
[info]   1      8...", but got "1       10      val[1b  8       16
[info]   1      10      val3b   8       NULL]
[info]   1      10      val3a   6       12
[info]   1      8..." Result did not match for query #4
[info]   SELECT    Count(DISTINCT(t1a)),
[info]             t1b,
[info]             t3a,
[info]             t3b,
[info]             t3c
[info]   FROM      t1 natural left JOIN t3
[info]   WHERE     t1a IN
[info]             (
[info]                    SELECT t2a
[info]                    FROM   t2
[info]                    WHERE t1d = t2d)
[info]   AND       t1b > t3b
[info]   GROUP BY  t1a,
[info]             t1b,
[info]             t3a,
[info]             t3b,
[info]             t3c
[info]   ORDER BY  t1a DESC, t3b DESC (SQLQueryTestSuite.scala:383)
[info]   org.scalatest.exceptions.TestFailedException:

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. It seems that we are hitting the corner case because the query has a sort on a subset of columns.

    def isSorted(plan: LogicalPlan): Boolean = plan match {
      case _: Join | _: Aggregate | _: Generate | _: Sample | _: Distinct => false
      case _: DescribeCommandBase
          | _: DescribeColumnCommand
          | _: DescribeTableStatement
          | _: DescribeColumnStatement => true
      case PhysicalOperation(_, _, Sort(_, true, _)) => true
      case _ => plan.children.iterator.exists(isSorted)
    }

1 10 val3a 6 12
1 8 val3a 6 12
1 8 val3a 6 12
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-join.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3023,9 +3023,9 @@ select udf(b.unique1) from
-- !query 147 schema
struct<CAST(udf(cast(unique1 as string)) AS INT):int>
-- !query 147 output
0
NULL
NULL
0
NULL
NULL

Expand Down
10 changes: 5 additions & 5 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.util.{Locale, TimeZone}

import scala.util.control.NonFatal

import org.apache.spark.SparkException
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.plans.logical.sql.{DescribeColumnStatement, DescribeTableStatement}
Expand Down Expand Up @@ -133,6 +133,10 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
private val notIncludedMsg = "[not included in comparison]"
private val clsName = this.getClass.getCanonicalName

override def sparkConf: SparkConf = super.sparkConf
// Fewer shuffle partitions to speed up testing.
.set(SQLConf.SHUFFLE_PARTITIONS, 5)

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Python UDF test, this seems to increase from 4 to 5. Did I understand correctly?

@HyukjinKwon HyukjinKwon Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually why don't we try 4 @wangyum?

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for 4.

@wangyum wangyum Sep 22, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I'll try to set it to 4. This is because it is set to 5 in two places:

spark/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala

Lines 58 to 64 in 359375e

/**
* A map used to store all confs that need to be overridden in sql/core unit tests.
*/
val overrideConfs: Map[String, String] =
Map(
// Fewer shuffle partitions to speed up testing.
SQLConf.SHUFFLE_PARTITIONS.key -> "5")

spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala

Lines 613 to 620 in 42b80ae

/**
* A map used to store all confs that need to be overridden in sql/hive unit tests.
*/
val overrideConfs: Map[String, String] =
Map(
// Fewer shuffle partitions to speed up testing.
SQLConf.SHUFFLE_PARTITIONS.key -> "5"
)

@wangyum wangyum Sep 23, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=4 has another issue: --SET spark.sql.autoBroadcastJoinThreshold=10485760 will success, but --SET spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=true will failed:

22:31:31.233 ERROR org.apache.spark.sql.SQLQueryTestSuite: Error using configs: spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=true,spark.sql.codegen.wholeStage=true,spark.sql.codegen.factoryMode=CODEGEN_ONLY
[info] - subquery/in-subquery/not-in-joins.sql *** FAILED *** (32 seconds, 609 milliseconds)
[info]   subquery/in-subquery/not-in-joins.sql
[info]   Expected "1    16      12      [21
[info]   1      16      12      10]
[info]   1      10      NULL    12
[info]   1      6       8       ...", but got "1        16      12      [10
[info]   1      16      12      21]
[info]   1      10      NULL    12
[info]   1      6       8       ..." Result did not match for query #6
[info]   SELECT Count(DISTINCT( t1a )),
[info]          t1b,
[info]          t1c,
[info]          t1d
[info]   FROM   t1
[info]   WHERE  t1a NOT IN (SELECT t2a
[info]                      FROM   t2
[info]                      JOIN t1
[info]                      WHERE  t2b <> t1b)
[info]   GROUP  BY t1b,
[info]             t1c,
[info]             t1d
[info]   HAVING t1d NOT IN (SELECT t2d
[info]                      FROM   t2
[info]                      WHERE  t1d = t2d)
[info]   ORDER BY t1b DESC (SQLQueryTestSuite.scala:383)

@cloud-fan cloud-fan Sep 23, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add an ORDER BY to make the query output deterministic?


/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
"blacklist.sql" // Do NOT remove this one. It is here to test the blacklist functionality.
Expand Down Expand Up @@ -291,10 +295,6 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {

testCase match {
case udfTestCase: UDFTest =>
// In Python UDF tests, the number of shuffle partitions matters considerably in
// the testing time because it requires to fork and communicate between external
// processes.
localSparkSession.conf.set(SQLConf.SHUFFLE_PARTITIONS.key, 4)
registerTestUDF(udfTestCase.udf, localSparkSession)
case _ =>
}
Expand Down