[SPARK-19426][SQL] Custom coalesce for Dataset #16766

mariusvniekerk · 2017-02-01T16:39:56Z

What changes were proposed in this pull request?

This adds support for using the PartitionCoalescer features added in #11865 (SPARK-14042) to the Dataset API

How was this patch tested?

Manual tests

gatorsmile · 2017-02-02T07:33:34Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

Could you follow the styles documented in https://github.com/databricks/scala-style-guide?

gatorsmile · 2017-02-02T07:46:55Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

CoalesceLogical -> Coalesce?

Main reason is there was already a Coalesce expression class

gatorsmile · 2017-02-02T07:48:49Z

Could you please also add a few test cases? For example, DataFrameSuite or DatasetSuite.

felixcheung · 2017-02-02T17:41:11Z

I'd second that. I'd be interested to know if this implementation changes behavior for coalesce

gatorsmile · 2017-02-03T06:56:49Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

Please also add the function description, like what we did in the other functions in Dataset.scala?

gatorsmile · 2017-02-03T06:57:21Z

sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala

The same indent issue here.

case class CoalesceExec( numPartitions: Int, child: SparkPlan, partitionCoalescer: Option[PartitionCoalescer]) extends UnaryExecNode {

Do you guys have a .scalafmt.conf that applies all of this? that should make things cleaner.

gatorsmile · 2017-02-03T07:05:39Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

gatorsmile · 2017-02-03T07:12:05Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

The name still looks inconsistent with the others. How about PartitionCoalesce?

that sounds good

gatorsmile · 2017-02-03T07:12:24Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

This description is not right.

gatorsmile · 2017-02-03T07:12:44Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

Style issues:

case class PartitionCoalesce( numPartitions: Int, partitionCoalescer: Option[PartitionCoalescer], child: LogicalPlan) extends UnaryNode {

gatorsmile · 2017-02-03T07:14:22Z

sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala

Add it back?

gatorsmile · 2017-02-03T07:19:33Z

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala

cannot resolve '`a`' given input columns: [_c0, _c1];

Oh right csv doesn't do headers.

gatorsmile · 2017-02-03T07:23:48Z

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala

use WithTempPath to generate the path?

gatorsmile · 2017-02-03T07:25:44Z

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala

No need to do it, if you use withTempPath. This is an example

ah thanks. I looked at the writer tests

gatorsmile · 2017-02-03T07:26:14Z

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala

Nit: Remove this empty line.

mariusvniekerk · 2017-02-03T16:23:50Z

@felixcheung This does not touch any of the coalesce internals. Only allows setting a partitionCoalescer similar to what is already available in rdd.coalesce

gatorsmile · 2017-06-13T06:23:19Z

Sorry for the late reply. @mariusvniekerk

Could you please update the PR?

gatorsmile · 2017-06-13T06:23:28Z

ok to test

SparkQA · 2017-06-13T06:29:11Z

Test build #77969 has finished for PR 16766 at commit d4bde0b.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

mariusvniekerk · 2017-06-13T23:14:05Z

Let me rebase this. I don't currently have a clean way of testing this on Windows

gatorsmile · 2017-06-14T06:43:25Z

Could you run the following four commands to do a local test in your local environment?

dev/lint-scala
build/sbt -Phive hive/test
build/sbt sql/test
build/sbt catalyst/test

SparkQA · 2017-06-17T15:19:11Z

Test build #78212 has finished for PR 16766 at commit 00b2a7a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CoalesceExec(

SparkQA · 2017-06-17T15:48:47Z

Test build #78213 has finished for PR 16766 at commit e99cebc.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-17T22:09:27Z

Test build #78218 has finished for PR 16766 at commit 1d1b2fa.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DataSetSizeBasedPartitionCoalescer(maxSize: Int) extends

HyukjinKwon

Actual javadoc errors are as below:

[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/Dataset.java:2222: error: reference not found
[error]    * A {@link PartitionCoalescer} can also be supplied allowing the behavior of the partitioning to be
[error]               ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/Dataset.java:2223: error: reference not found
[error]    * customized similar to {@link RDD.coalesce}.
[error]

HyukjinKwon · 2017-06-19T04:31:10Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

   * the current partitioning is).
   *
+   * A [[PartitionCoalescer]] can also be supplied allowing the behavior of the partitioning to be
+   * customized similar to [[RDD.coalesce]].


I think it should be [[org.apache.spark.rdd.RDD##coalesce]].

HyukjinKwon · 2017-06-19T04:32:08Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

   * current upstream partitions will be executed in parallel (per whatever
   * the current partitioning is).
   *
+   * A [[PartitionCoalescer]] can also be supplied allowing the behavior of the partitioning to be


Sounds this trait is unable to be generated as is in Java. Simply wrapping `...` would be fine.

HyukjinKwon · 2017-07-24T03:22:55Z

Hi @mariusvniekerk, would you be able to fix the javadoc errors?

gatorsmile · 2017-08-06T06:12:41Z

cc @maropu Do you want to take this over?

maropu · 2017-08-06T07:52:47Z

@gatorsmile Sure! I'll do, Thanks!

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closes apache#18585 - SPARK-21359 Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closes apache#18667 - Fix the simpleString used in error messages Closes apache#18782 - Branch 2.1 Added: Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closes apache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18780 from HyukjinKwon/close-prs.

gatorsmile reviewed Feb 2, 2017

View reviewed changes

gatorsmile reviewed Feb 3, 2017

View reviewed changes

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala Outdated

Copy link

Member

gatorsmile Feb 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useless?

gatorsmile reviewed Feb 3, 2017

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala Outdated

Copy link

Member

gatorsmile Feb 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add it back?

gatorsmile reviewed Feb 3, 2017

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala Outdated

Copy link

Member

gatorsmile Feb 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Remove this empty line.

mariusvniekerk added 4 commits June 17, 2017 11:12

custom coalesce

0f1060c

Adapted unit test for custom coalesce from the one in RDDSuite

2742457

Addressing review comments.

2499364

Indentation fixup

00b2a7a

mariusvniekerk force-pushed the wip_customCoalesce branch from d4bde0b to 00b2a7a Compare June 17, 2017 15:15

Fixed style issues

e99cebc

Fixed failing tests

1d1b2fa

HyukjinKwon reviewed Jun 19, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Jul 31, 2017

[INFRA] Close stale PRs #18780

Closed

asfgit closed this in 3a45c7f Aug 5, 2017

maropu mentioned this pull request Aug 6, 2017

[SPARK-19426][SQL] Custom coalescer for Dataset #18861

Closed

[SPARK-19426][SQL] Custom coalesce for Dataset #16766

[SPARK-19426][SQL] Custom coalesce for Dataset #16766

Uh oh!

Conversation

mariusvniekerk commented Feb 1, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Feb 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented Feb 2, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Feb 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Feb 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariusvniekerk commented Feb 3, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

mariusvniekerk commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 14, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Feb 2, 2017 •

edited

Loading

gatorsmile Feb 3, 2017 •

edited

Loading

gatorsmile Feb 3, 2017 •

edited

Loading