Skip to content

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 6 commits into from
Closed

[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite #25891

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3dc0124
Reduce shuffle partitions to reduce testing time
wangyum Sep 22, 2019
6ec9761
Avoid test error in StreamSuite
wangyum Sep 22, 2019
55004b9
SHUFFLE_PARTITIONS to 4
wangyum Sep 22, 2019
b4f2d19
SHUFFLE_PARTITIONS to 5
wangyum Sep 23, 2019
ad6bee7
SHUFFLE_PARTITIONS to 4 and add sort
wangyum Sep 23, 2019
df51b69
DESC -> ASC
wangyum Sep 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ ORDER BY t1a DESC, t3b DESC
-- !query 4 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t3a:string,t3b:smallint,t3c:int>
-- !query 4 output
1 10 val3b 8 NULL
1 10 val1b 8 16
1 10 val3b 8 NULL

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, do we really need this?

@wangyum wangyum Sep 22, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

[info] - subquery/in-subquery/in-joins.sql *** FAILED *** (15 seconds, 359 milliseconds)
[info]   subquery/in-subquery/in-joins.sql
[info]   Expected "1    10      val[3b  8       NULL
[info]   1      10      val1b   8       16]
[info]   1      10      val3a   6       12
[info]   1      8...", but got "1       10      val[1b  8       16
[info]   1      10      val3b   8       NULL]
[info]   1      10      val3a   6       12
[info]   1      8..." Result did not match for query #4
[info]   SELECT    Count(DISTINCT(t1a)),
[info]             t1b,
[info]             t3a,
[info]             t3b,
[info]             t3c
[info]   FROM      t1 natural left JOIN t3
[info]   WHERE     t1a IN
[info]             (
[info]                    SELECT t2a
[info]                    FROM   t2
[info]                    WHERE t1d = t2d)
[info]   AND       t1b > t3b
[info]   GROUP BY  t1a,
[info]             t1b,
[info]             t3a,
[info]             t3b,
[info]             t3c
[info]   ORDER BY  t1a DESC, t3b DESC (SQLQueryTestSuite.scala:383)
[info]   org.scalatest.exceptions.TestFailedException:

@dongjoon-hyun dongjoon-hyun Sep 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. It seems that we are hitting the corner case because the query has a sort on a subset of columns.

    def isSorted(plan: LogicalPlan): Boolean = plan match {
      case _: Join | _: Aggregate | _: Generate | _: Sample | _: Distinct => false
      case _: DescribeCommandBase
          | _: DescribeColumnCommand
          | _: DescribeTableStatement
          | _: DescribeColumnStatement => true
      case PhysicalOperation(_, _, Sort(_, true, _)) => true
      case _ => plan.children.iterator.exists(isSorted)
    }

1 10 val3a 6 12
1 8 val3a 6 12
1 8 val3a 6 12
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,8 @@ ORDER BY t1b DESC
-- !query 6 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t1c:int,t1d:bigint>
-- !query 6 output
1 16 12 10
1 16 12 21
1 16 12 10
1 10 NULL 12
1 6 8 10
1 NULL 16 22
Expand Down
10 changes: 5 additions & 5 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.util.{Locale, TimeZone}

import scala.util.control.NonFatal

import org.apache.spark.SparkException
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.plans.logical.sql.{DescribeColumnStatement, DescribeTableStatement}
Expand Down Expand Up @@ -133,6 +133,10 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
private val notIncludedMsg = "[not included in comparison]"
private val clsName = this.getClass.getCanonicalName

override def sparkConf: SparkConf = super.sparkConf
// Fewer shuffle partitions to speed up testing.
.set(SQLConf.SHUFFLE_PARTITIONS, 4)

/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
"blacklist.sql" // Do NOT remove this one. It is here to test the blacklist functionality.
Expand Down Expand Up @@ -291,10 +295,6 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {

testCase match {
case udfTestCase: UDFTest =>
// In Python UDF tests, the number of shuffle partitions matters considerably in
// the testing time because it requires to fork and communicate between external
// processes.
localSparkSession.conf.set(SQLConf.SHUFFLE_PARTITIONS.key, 4)
registerTestUDF(udfTestCase.udf, localSparkSession)
case _ =>
}
Expand Down